Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ModelCheckpoint Filepath Doesn't Use Logger Save Dir #734

Closed
collinmccarthy opened this issue Jan 23, 2020 · 4 comments · Fixed by #771
Closed

ModelCheckpoint Filepath Doesn't Use Logger Save Dir #734

collinmccarthy opened this issue Jan 23, 2020 · 4 comments · Fixed by #771
Labels
bug Something isn't working good first issue Good for newcomers help wanted Open to be worked on
Milestone

Comments

@collinmccarthy
Copy link

🐛 Bug

Not sure if this is intended, but the model checkpoint isn't using the same directory as the logger, even if the logger exists. I would have expected this line here to be self.logger.save_dir instead of self.default_save_path.

Thank you,
-Collin

@collinmccarthy collinmccarthy added the bug Something isn't working label Jan 23, 2020
@bosbes
Copy link

bosbes commented Jan 23, 2020

I am running into the same issue and feel like I must be doing something wrong. If I do not define a logger and checkpoint callback, the logger and checkpoint are saved in a clear structure under lightning_logs. But if I do provide them, it turns out to be difficult to get Lightning to follow that same structure. If I use lightning_logs as filepath I get the warning that that directory already exists and that my existing checkpoints will be overwritten.

@Borda Borda added good first issue Good for newcomers help wanted Open to be worked on need fix labels Jan 24, 2020
@sneiman
Copy link
Contributor

sneiman commented Jan 26, 2020

Were you able to find the checkpoints? I cant seem to find them anywhere!

@collinmccarthy
Copy link
Author

The checkpoints are currently being saved to a subfolder of the current working directory. If you print the current working directory at the top of our file with print(os.getcwd()) that should tell you where to find them.

@sneiman
Copy link
Contributor

sneiman commented Jan 27, 2020

FWIW, using the default checkpoint_callback, the checkpoint is not in default location. Still looking for it! Though no more default checkpoint_callback for me ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers help wanted Open to be worked on
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants