Linear warmup sched

[ananyahjha93]

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

Fixes # (issue).

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[pep8speaks]

Hello @ananyahjha93! Thanks for updating this PR.

• In the file pl_bolts/optimizers/lr_scheduler.py:

Line 100:80: W504 line break after binary operator
Line 101:72: W504 line break after binary operator
Line 102:84: W504 line break after binary operator
Line 103:72: W504 line break after binary operator

Comment last updated at 2020-08-05 17:59:48 UTC

[ananyahjha93]

this comes from earlier versions of pytorch where scheduler.step() was called before the epoch instead of after.

[justusschock]

but do we still support these old versions?

[ananyahjha93]

@justusschock current version of pytorch still defines _test_against_closed_form() for now, all we need to do is define that function to support the old version. I have a warning and doc available for this

[codecov]

Codecov Report

Merging #138 into master will increase coverage by 0.17%.
The diff coverage is 93.33%.

@@            Coverage Diff             @@
##           master     #138      +/-   ##
==========================================
+ Coverage   91.14%   91.32%   +0.17%     
==========================================
  Files          82       86       +4     
  Lines        4056     4172     +116     
==========================================
+ Hits         3697     3810     +113     
- Misses        359      362       +3     

Flag	Coverage Δ
#unittests	91.32% <93.33%> (+0.17%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pl_bolts/optimizers/lr_scheduler.py	93.10% <93.10%> (ø)
pl_bolts/optimizers/__init__.py	100.00% <100.00%> (ø)
pl_bolts/models/rl/per_dqn_model.py	80.00% <0.00%> (-0.49%)	⬇️
...bolts/models/self_supervised/amdim/ssl_datasets.py	75.36% <0.00%> (-0.36%)	⬇️
pl_bolts/models/rl/dqn_model.py	82.17% <0.00%> (-0.18%)	⬇️
.../models/autoencoders/basic_vae/basic_vae_module.py	92.78% <0.00%> (-0.15%)	⬇️
..._bolts/models/self_supervised/moco/moco2_module.py	85.32% <0.00%> (-0.14%)	⬇️
pl_bolts/models/self_supervised/cpc/cpc_module.py	80.82% <0.00%> (-0.14%)	⬇️
...lts/models/self_supervised/simclr/simclr_module.py	91.05% <0.00%> (-0.08%)	⬇️
pl_bolts/models/self_supervised/cpc/networks.py	94.59% <0.00%> (-0.05%)	⬇️
... and 19 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ac363dd...ce206aa. Read the comment docs.

[ananyahjha93]

@Borda feedback on formatting as well, used https://github.com/psf/black on @nateraw 's advice.

[justusschock]

I think there is an epsilon defined somewhere in lightning iirc, maybe use this to avoid duplication?

cc @Borda who might know, where it is ^^

[ananyahjha93]

@Borda so I had this question about default epsilons in lightning, numpy.finfo(numpy.float32).eps is 1.1920929e-07. Since learning rates used are up to 1e-6, we need a higher precision in schedulers. But, numpy.finfo(numpy.float64).eps comes out to be 2.220446049250313e-16, which results in failing tests. So the ideal epsilon seemed 1e-12.

[justusschock]

but do we still support these old versions?