Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

graceful training cleanup after Keyboard Interrupt #856

Merged

Conversation

jeremyjordan
Copy link
Contributor

@jeremyjordan jeremyjordan commented Feb 15, 2020

Before submitting

  • Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
  • Did you read the contributor guideline?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

What does this PR do?

Fixes #795

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

@jeremyjordan
Copy link
Contributor Author

We should be able to close #657 as well

@Borda Borda added the feature Is an improvement or enhancement label Feb 15, 2020
@Borda Borda added this to the 0.6.1 milestone Feb 15, 2020
Copy link
Member

@Borda Borda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall the run_training_teardown() be called also after training?

pytorch_lightning/trainer/trainer.py Show resolved Hide resolved
pytorch_lightning/trainer/training_loop.py Show resolved Hide resolved
pytorch_lightning/trainer/training_loop.py Show resolved Hide resolved
pytorch_lightning/trainer/training_loop.py Show resolved Hide resolved
pytorch_lightning/trainer/training_loop.py Show resolved Hide resolved
@jeremyjordan
Copy link
Contributor Author

@Borda updated to fix merge conflicts with master. were there any more changes requested?

Copy link
Member

@Borda Borda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@Borda
Copy link
Member

Borda commented Feb 17, 2020

@jeremyjordan just fixed another conflict, pls check it...

@jeremyjordan
Copy link
Contributor Author

@Borda looks good! thanks for addressing that

@Borda Borda added the ready PRs ready to be merged label Feb 17, 2020
@Borda Borda mentioned this pull request Feb 18, 2020
@williamFalcon
Copy link
Contributor

@jeremyjordan rebase?

@jeremyjordan
Copy link
Contributor Author

jeremyjordan commented Feb 22, 2020

@williamFalcon rebased and ready for review! unfortunately the diff renders poorly

@williamFalcon williamFalcon merged commit e05586c into Lightning-AI:master Feb 22, 2020
@ethanwharris ethanwharris mentioned this pull request Feb 25, 2020
4 tasks
@Borda Borda changed the title graceful training cleanup after KeyboardInterrupt graceful training cleanup after Keyboard Interrupt Mar 8, 2020
tullie pushed a commit to tullie/pytorch-lightning that referenced this pull request Apr 3, 2020
…ning-AI#856)

Co-authored-by: Jirka Borovec <Borda@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Is an improvement or enhancement ready PRs ready to be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

support graceful training cleanup after Keyboard Interrupt
3 participants