[WIP] Hypertuner class

[SkafteNicki]

What does this PR do?

This is redo of PR #1998. The last PR was too big land because it was a major refactoring, so I will try to split into more manageable pieces.

This PR is basically how the core structure of the HyperTuner class should look. It does not implement any functionalities yet. After this, I plan on 3 follow up PRs, one for each of the features that the HyperTuner class will include:

• Lr finder (deprecate from trainer, add more tests, get working in ddp mode, add support for DataModule)
• Batch size finder (deprecate from trainer, somewhat stable at this point, add support for DataModule)
• new: n worker searcher Let's add a suggested_num_workers() method? #2196) (already have some code, just need to polish for the new api)

Tagging for input: @Borda , @awaelchli , @justusschock

Question: We have informed user that from v0.9 no more API changes would happen (before v1.0). This is a somewhat big API change, so to keep that promise, I see two options:

• Wait with this to after v1.0
• Let the hypertuner class be a functionality of bolts, and deprecate the lr finder and batch scaler in lightning

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?
• Did you verify new and existing tests pass locally with your changes?
• If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[williamFalcon]

@SkafteNicki please wait a week... we need to finish refactors.

We can decide if it's before or after 0.9 then and if it stays here or bolts

[SkafteNicki]

@williamFalcon completly fine with me, whatever makes most sense for the project :]

[rohitgr7]

@SkafteNicki does it support user-defined callbacks to be used when calling lr_find or auto_scale_batch_size? as of now, it drops them.

[SkafteNicki]

Based on discussion the tuner class is being dropped for now, instead we will refactor tuner methods into a trainer.tune() method. Closing this.

[Borda]

Based on discussion the tuner class is being dropped for now, instead we will refactor tuner methods into a trainer.tune() method. Closing this.

Anyway, GREAT work on this PR! ❤️

[jcsagar]

So with the new datamodule, is there no way to tune batch_size? I was really excited by the Hypertuner class in this thread, so am a little bummed this is closed for now.

For my current workflow, I tried to pass in trainer.tune(lightningmodule, datamodule) with auto_scale_batch_size. It tries to scale the batch size, (incorrectly) succeeds with every attempt, until it finds the batch_size of 2^26 (after 25 trials). Which it tries to then fit and fails. I looked into why this was happening and realized that datamodule isn't actually looked at by the scale_batch_size function in training_tricks.py.

[awaelchli]

@jcsagar Fixed in #3266 and #3271

[jcsagar]

@awaelchli Updated to master and works well. Great stuff, thanks!!!