Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KeyError: 'early_stop_callback_wait' when calling training with auto_lr_find #1650

Closed
Diyago opened this issue Apr 28, 2020 · 1 comment · Fixed by #1676
Closed

KeyError: 'early_stop_callback_wait' when calling training with auto_lr_find #1650

Diyago opened this issue Apr 28, 2020 · 1 comment · Fixed by #1676
Labels
bug Something isn't working help wanted Open to be worked on

Comments

@Diyago
Copy link

Diyago commented Apr 28, 2020

🐛 Bug

Set auto_lr_find = True and try to train. Training with false flag works well.

To Reproduce

Steps to reproduce the behavior:

model = LightningCanserClassifier(config)
trainer = pl.Trainer(logger=tb_logger, early_stop_callback=early_stop_callback,
                             checkpoint_callback=checkpoint_callback, **config['training']['Trainer'])
trainer.fit(model, train_dataloader=train_loader, val_dataloaders=valid_loader)
 /home/dex/anaconda3/bin/python "/home/dex/Desktop/panda canser kaggle/train.py"
INFO:lightning:GPU available: True, used: True
INFO:lightning:CUDA_VISIBLE_DEVICES: [0]
Finding best initial lr: 100%|██████████| 100/100 [00:35<00:00,  2.72it/s]Traceback (most recent call last):
  File "/home/dex/Desktop/panda canser kaggle/train.py", line 144, in <module>
    trainer.fit(model, train_dataloader=train_loader, val_dataloaders=valid_loader)
  File "/home/dex/anaconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 734, in fit
    self._run_lr_finder_internally(model)
  File "/home/dex/anaconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/lr_finder.py", line 31, in _run_lr_finder_internally
    lr_finder = self.lr_find(model)
  File "/home/dex/anaconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/lr_finder.py", line 164, in lr_find
    self.restore(str(save_path), on_gpu=self.on_gpu)
  File "/home/dex/anaconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/training_io.py", line 289, in restore
    self.restore_training_state(checkpoint)
  File "/home/dex/anaconda3/lib/python3.7/site-packages/pytorch_lightning/trainer/training_io.py", line 372, in restore_training_state
    self.early_stop_callback.wait = checkpoint['early_stop_callback_wait']
KeyError: 'early_stop_callback_wait'

Expected behavior

Environment

Please copy and paste the output from our
environment collection script
(or fill out the checklist below manually).

You can get the script and run it with:

wget https://github.com/raw/PyTorchLightning/pytorch-lightning/master/tests/collect_env_details.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
  • PyTorch Version (e.g., 1.0): 1.3.1
  • OS (e.g., Linux): ubuntu
  • How you installed PyTorch (conda, pip, source): conda
  • Build command you used (if compiling from source):
  • Python version: 3.7
  • CUDA/cuDNN version: cuda10.0.130_cudnn7.6.3_0
  • GPU models and configuration: 1070ti
  • Any other relevant information:
@Diyago Diyago added bug Something isn't working help wanted Open to be worked on labels Apr 28, 2020
@github-actions
Copy link
Contributor

Hi! thanks for your contribution!, great first issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Open to be worked on
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant