Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LightningModule.load_from_checkpoint not working with .ckpt from 0.7.6 #2334

Closed
dscarmo opened this issue Jun 23, 2020 · 2 comments · Fixed by #2405
Closed

LightningModule.load_from_checkpoint not working with .ckpt from 0.7.6 #2334

dscarmo opened this issue Jun 23, 2020 · 2 comments · Fixed by #2405
Assignees
Labels
bug Something isn't working help wanted Open to be worked on priority: 0 High priority task
Milestone

Comments

@dscarmo
Copy link
Contributor

dscarmo commented Jun 23, 2020

🐛 Bug

Trying to use an old experiment .ckpt (generated with ModelCheckpoint(monitor="val_loss", mode="min') in 0.7.6) results in an error when trying to load.

To Reproduce

Steps to reproduce the behavior:

  1. Train something with 0.7.6 and save checkpoint with the checkpoint callback
  2. Try to load checkpoint with 0.8.1
Trying to load: ckpts/MNIST_0.7.6epoch=4.ckpt with PL version: 0.8.1
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-36a7290b43c6> in <module>()
     35 selected_ckpt = glob(os.path.join("ckpts", "*.ckpt"))[0]
     36 print(f"Trying to load: {selected_ckpt} with PL version: {pl.__version__}")
---> 37 LitClassifier.load_from_checkpoint(selected_ckpt)

1 frames
/usr/local/lib/python3.6/dist-packages/pytorch_lightning/core/saving.py in load_from_checkpoint(cls, checkpoint_path, map_location, hparams_file, tags_csv, *args, **kwargs)
    169         checkpoint[cls.CHECKPOINT_HYPER_PARAMS_KEY].update(kwargs)
    170 
--> 171         model = cls._load_model_state(checkpoint, *args, **kwargs)
    172         return model
    173 

/usr/local/lib/python3.6/dist-packages/pytorch_lightning/core/saving.py in _load_model_state(cls, checkpoint, *args, **kwargs)
    184 
    185             if cls.CHECKPOINT_HYPER_PARAMS_TYPE in checkpoint:
--> 186                 model_args = checkpoint[cls.CHECKPOINT_HYPER_PARAMS_TYPE](model_args)
    187 
    188             args_name = checkpoint.get(cls.CHECKPOINT_HYPER_PARAMS_NAME)

TypeError: 'str' object is not callable

Here is a Colab notebook reproducing the issue: https://colab.research.google.com/drive/1s7SzG3EkLJZOmUOkb0mJ85zDcOxBcDLG?usp=sharing

Maybe this is related to the changes in hparams recently?
It happened in my local enviroment and in Google Colab's enviroment.
I was able to load the checkpoint after downgrading to 0.7.6 without problems.

Expected behavior

Being able to use old .ckpts.

Environment

  • PyTorch Version (e.g., 1.0): 1.5
  • OS (e.g., Linux): Ubuntu 18.04
  • How you installed PyTorch (conda, pip, source): pip
  • Build command you used (if compiling from source):
  • Python version: 3.6.9
  • CUDA/cuDNN version: 10.2
  • GPU models and configuration: 1060
  • Any other relevant information:

Also happened in the Colab enviroment.

@dscarmo dscarmo added bug Something isn't working help wanted Open to be worked on labels Jun 23, 2020
@Borda Borda added the priority: 0 High priority task label Jun 24, 2020
@dscarmo dscarmo changed the title LightningModule.load_from_checkpoint not working with .ckpt from 0.7 LightningModule.load_from_checkpoint not working with .ckpt from 0.7.6 Jun 25, 2020
@dscarmo
Copy link
Contributor Author

dscarmo commented Jun 25, 2020

Added a Google Colab notebook reproducing the issue.

@williamFalcon williamFalcon added this to the 0.8.x milestone Jun 26, 2020
Borda added a commit that referenced this issue Jun 28, 2020
@Borda Borda mentioned this issue Jun 28, 2020
7 tasks
williamFalcon pushed a commit that referenced this issue Jun 28, 2020
@williamFalcon
Copy link
Contributor

image

@dscarmo no more bug :) @Borda fixed it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Open to be worked on priority: 0 High priority task
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants