nan detection and intervention #1097
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
awaelchli
changed the title
added 2 commits
March 9, 2020 00:49
|
hey there, we have added GPU CI test, so could we kindly ask to rebase/merge master which will trigger these tests so we do not need to test it manually... Thx for your understanding 🤖 |
|
Hello @awaelchli! Thanks for updating this PR. There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻 Comment last updated at 2020-03-18 12:37:26 UTC |
Codecov Report
@@ Coverage Diff @@
## master #1097 +/- ##
======================================
Coverage 89% 89%
======================================
Files 62 62
Lines 3164 3173 +9
======================================
+ Hits 2810 2820 +10
+ Misses 354 353 -1 |
added 2 commits
March 18, 2020 12:03
awaelchli
changed the title
|
I think this is close to finished, just need review and core developers to comment/decide about the open question in the description above. Thanks. |
Borda
requested changes
Mar 18, 2020
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>
Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com>
Borda
added
bug
tullie
pushed a commit
to tullie/pytorch-lightning
that referenced
this pull request
Apr 3, 2020
* check for nan values * test nan detection on loss * sys.exit * whitespace * detect nan and inf values in loss and params * update * added documentation * moved detect nan to training loop, remove flag for print * blank line * test * rename * deprecate print_nan_grads * deprecated print_nan_grads * remove unused imports * update changelog * fix line too long * correct deprecated version Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com> * raise exception instead of sysexit Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com> * raise exception instead of sysexit Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com> * Update pytorch_lightning/trainer/training_tricks.py Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com> * Update pytorch_lightning/trainer/training_tricks.py Co-Authored-By: Jirka Borovec <Borda@users.noreply.github.com> * fix test Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before submitting
What does this PR do?
Fixes #1008.
Open question: what happens to the
print_nan_gradsTrainer argument? is it still useful?print_nan_gradsargument and print them whenever nan is detected. Since training stops anyway, it wouldn't make sense to have this arg.PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃