Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Propose to save new model before deleting previous ones in ModelCheckpointing #1150

Closed
Ir1d opened this issue Mar 14, 2020 · 3 comments · Fixed by #1453
Closed

Propose to save new model before deleting previous ones in ModelCheckpointing #1150

Ir1d opened this issue Mar 14, 2020 · 3 comments · Fixed by #1453
Assignees
@Ir1d
Labels
feature Is an improvement or enhancement help wanted Open to be worked on let's do it! approved to implement
Milestone
pl:0.7.x

Comments

@Ir1d
Copy link
Contributor

Ir1d commented Mar 14, 2020 •
edited by Borda
Loading

🚀 Feature

In an edge case, the trainer deleted previous model and then was killed because of system error before successfully saving new model. Thus all the models were lost.
I understand specifying save_top_k > 1 helps, and saving before deleting leads to larger disk consumption. But it might be good to provide an option for this?

Motivation

Pitch

in the worst case, you have two but never none...

Alternatives

Additional context
@Ir1d Ir1d added feature Is an improvement or enhancement help wanted Open to be worked on labels Mar 14, 2020
@Borda Borda added this to the 0.7.2 milestone Mar 16, 2020
@Borda Borda added the let's do it! approved to implement label Mar 16, 2020
@Borda
Copy link
Member

Borda commented Mar 16, 2020

That is good point, I like it... @PyTorchLightning/core-contributors ^^

@williamFalcon
Copy link
Contributor

@Ir1d sounds great. submit a PR?

@williamFalcon williamFalcon modified the milestones: 0.7.2, 0.7.3 Apr 7, 2020
@Ir1d
Copy link
Contributor Author

Ir1d commented Apr 7, 2020

I can do it this friday

@Ir1d Ir1d mentioned this issue Apr 11, 2020
Merged
5 tasks
@mergify mergify bot closed this as completed in #1453 Apr 16, 2020
@Borda Borda modified the milestones: 0.7.4, v0.7.x Apr 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Ir1d Ir1d

Labels
feature Is an improvement or enhancement help wanted Open to be worked on let's do it! approved to implement
Projects
None yet
Milestone
pl:0.7.x
Development

Successfully merging a pull request may close this issue.

feat: save checkpoint before deleting old ones Ir1d/pytorch-lightning
3 participants
@williamFalcon @Borda @Ir1d