Cosine learning rate schedule - minimum learning rate

[RicardoDominguez]

Regarding cosine learning rate decay, in the literature the learning rate is typically decayed to some percentage of the peak learning rate rather than decayed to 0 (e.g., Llama 2 decays the final learning rate down to 10% of the peak learning rate).

I am unsure if axolotl currently offers a straightforward way of setting the minimum learning rate when using cosine decay. Here is a simple implementation where the user simply needs to specify cosine_min_lr_ratio in the config file (e.g., cosine_min_lr_ratio=0.1 decays the final learning rate down to 10% of the peak learning rate).

[winglian]

This is great, I think the only thing we may want to do is warn if a user sets cosine_min_lr_ratio and also uses deepspeed, that cosine_min_lr_ratio COULD be ignored depending on whether the user set a scheduler in their deepspeed JSON. the example deepspeed files in axolotl don't have this set, so should be less of an issue if people use those.

[winglian]

@RicardoDominguez one last thing, could you update the README to include this option in the documented yaml options please?

[RicardoDominguez]

Great, thanks @winglian!