Checkpooint Callback not called when training GAN from documentation

[PasqualeZingo]

I tried to add a model checkpoint callback to the GAN example from the documentation (https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/domain_templates/gan.py), but it is failing silently; no checkpoints are being written. I can't find anything in the checkpoint docs that indicate why it is not working.

What's your environment?

• OS: Linux
• Packaging: Conda
• Version : 0.6, pytorch 1.4

[github-actions]

Hey, thanks for your contribution! Great first issue!

[jeremyjordan]

@PasqualeZingo could you reproduce the error in a Colab notebook and share the link?

[PasqualeZingo]

Sure thing, here it is: https://colab.research.google.com/drive/1wvZXP3WYxolfa4oA0mVnawtCXDr19eLd

Thanks @jeremyjordan !

[jeremyjordan]

Ah ok I think I know what's going on here.

If you look at the checkpointing callback, it's invoked with on_validation_end.

However, because you don't have a validation_step defined, the trainer is skipping validation. As a result, the callback to save the model checkpoints is never actually called.

[williamFalcon]

should we enable the checkpoint callback to use any metric across training or validation step?

[PasqualeZingo]

That would be ideal, in the mean time I can fool the validation loop into running for my use. Would it be worth updating the docs in the mean time to clarify about the checkpoint callback?

[jeremyjordan]

Yes, we should be running this on_epoch_end after both training and validation are complete.

[williamFalcon]

merged #1043