Adding support for the CosineAnnealingWarmRestarts scheduler

[Alicegaz]

Fixes #2849

Seems like there is currently no support for the schedulers taking values to the step function.
This commit adds support for any scheduler (not particularly ReduceOnPlateau scheduler) taking in validation metrics value/epoch/step_progress.

Step progress is a float value that CosineAnnealingWarmRestarts scheduler takes in cases when a step function is called after each step, which is found as current_epoch+batch_idx/num_training_batches.

Now, the 'monitor' attribute in the schedulers dictionary can also take strings 'epoch' and 'iter_frac' (denotes step progress). As there will be the need for other input values, the dictionary of additional 'monitor' keys and short lambda functions implementing their computation can be extended or moved to the utils section.

[pep8speaks]

Hello @Alicegaz! Thanks for updating this PR.

• In the file pytorch_lightning/trainer/training_loop.py:

Line 272:42: E127 continuation line over-indented for visual indent
Line 274:34: E127 continuation line over-indented for visual indent
Line 274:58: E225 missing whitespace around operator

Comment last updated at 2020-08-10 11:40:06 UTC

[codecov]

Codecov Report

Merging #2851 into master will decrease coverage by 55%.
The diff coverage is 20%.

@@           Coverage Diff            @@
##           master   #2851     +/-   ##
========================================
- Coverage      90%     36%    -55%     
========================================
  Files          78      78             
  Lines        7004    6927     -77     
========================================
- Hits         6321    2466   -3855     
- Misses        683    4461   +3778     

[SkafteNicki]

A few comment, PR is looking good.
Please also solve the pep issues.

[awaelchli]

@ananyahjha93 You might want to take a look at this since you recently added a scheduler to bolts.
Lightning-Universe/lightning-bolts#138

[mergify]

This pull request is now in conflict... :(

[ananyahjha93]

Blocking this PR for now. @Alicegaz This is still up for discussion: #2849

[ananyahjha93]

@Alicegaz closing this PR now, based on the discussions in #2849, we don't think this is the right way to go.

[edenlightning]

@ananyahjha93 what's the decision here?

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If you need further help see our docs: https://pytorch-lightning.readthedocs.io/en/latest/CONTRIBUTING.html#pull-request or ask the assistance of a core contributor here or on Slack. Thank you for your contributions.

[stale]

This pull request is going to be closed. Please feel free to reopen it create a new from the actual master.