Learning rate finder auto-suggest-LR algorithm is slightly too naive

[mstewart141]

🐛 Bug

The learning rate finder auto-suggest-LR finds the point of steepest loss descent but can be tricked by spikes early in the process. A short burn-in period at the beginning would resolve the issue.

To Reproduce

[Borda]

@SkafteNicki mind have a look ^^

[williamFalcon]

This makes sense... @mstewart141 mind submitting a PR? :)

[williamFalcon]

but wouldn't it also be tricked by spikes anywhere? we're talking about local mins here...

[justusschock]

Maybe something like a patients would be a good idea... If it doesn't furhter decrease after 5 additional lr changes or something, use the lr with the minimum so far...

[mstewart141]

i'm happy to try and help out.

one simple fix would be to add a "minimum_lr_threshold" kwarg to the plot function referenced above, with a default value of say 1e-5. few models in practice want a max/initial lr below that figure (of course the default could be even lower as well). then, when plotting, plot the whole plot as done now, but select the best suggestion in the range only after the min thresh.

the same fix could be applied to the suggestions that feed directly into the Trainer. the options would be to either pick a reasonable default and stick with the current api, or to accept Union[bool, float] for auto_lr_find, say, and interpret the float as the min threshold beyond which to consider suggestions.

[mstewart141]

but wouldn't it also be tricked by spikes anywhere? we're talking about local mins here...

i think that in practice the extreme spikes are a symptom of going from "totally random/untuned" to "ever so slightly tuned" and occur primarily right at the very very beginning.

of course, some models may behave more pathologically, but making a good suggestion for such models is probably out of scope for a simple LR suggester