Ranger — a synergistic optimizer combining RAdam (Rectified Adam) and Lookahead in one codebase.
RAdam stabilizes training at the start. (reference: "On the Variance of the Adaptive Learning Rate and Beyond" paper by Liyuan Liu et al.)
Lookahead stabilizes training and convergence during the rest of training. (reference: "Lookahead Optimizer: k steps forward, 1 step back" paper by Michael Zhang, Geoffrey Hinton, et al.)
Usage and notebook to test with coming shortly.
- Python 3.x
- PyTorch 1.x
Based on original work done by Less Wright (lessw2020). He built on LonePatient's implementation for the Lookahead part, and the official RAdam code for RAdam:
- Lookahead implementation from LonePatient - https://github.com/lonePatient/lookahead_pytorch/blob/master/optimizer.py
- RAdam code by https://github.com/LiyuanLucasLiu/RAdam/blob/master/radam.py