You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While deep learning models have replaced hand-designed features across many domains,
these models are still trained with hand-designed optimizers. In this work, we leverage the same
scaling approach behind the success of deep learning to learn versatile optimizers. We train an
optimizer for deep learning which is itself a small neural network that ingests gradients and
outputs parameter updates. Meta-trained with approximately four thousand TPU-months of
compute on a wide variety of optimization tasks, our optimizer not only exhibits compelling
performance, but optimizes in interesting and unexpected ways. It requires no hyperparameter
tuning, instead automatically adapting to the specifics of the problem being optimized. We open
source our learned optimizer, meta-training code, the associated train and test data, and an
extensive optimizer benchmark suite with baselines at velo-code.github.io.
actually, I read the paper and implementation before. However, I am not confident in implementing the VeLO optimizer effectively & Pytorch-friendly. But, I'll try!
https://arxiv.org/abs/2211.09760
https://github.com/google/learned_optimization/tree/main/learned_optimization/research/general_lopt
The text was updated successfully, but these errors were encountered: