[WIP] Replace grad norm. try to fix TPU dataloader issue

[srush]

Still playing around with TPU. I can now train one epoch fast, but when I start the second epoch it just ends with no error. Trying to figure out why that could be:

It seems to be something about this:

pytorch/xla#1191

Not sure if TPU dataloaders need to be specifically restarted?

[pep8speaks]

Hello @srush! Thanks for opening this PR.

• In the file pytorch_lightning/trainer/data_loading.py:

Line 179:13: W503 line break before binary operator
Line 193:13: W503 line break before binary operator
Line 194:13: E128 continuation line under-indented for visual indent
Line 194:13: W503 line break before binary operator

Do see the Hitchhiker's guide to code style

[williamFalcon]

@dlibenzi @mruberry any ideas?

[dlibenzi]

@dlibenzi @mruberry any ideas?

I've got no error, how can I have an idea? 😄
You are re-creating the ParallelLoader at every EPOCH, right?

[williamFalcon]

@dlibenzi nope! is that the issue?
Currently, we create it right before all the epochs.

https://github.com/PyTorchLightning/pytorch-lightning/blob/be244560b24b68b0236a4694707fb9bb63c2e6d0/pytorch_lightning/trainer/training_loop.py#L451-L455

[dlibenzi]

@dlibenzi nope! is that the issue?
Currently, we create it right before all the epochs.

https://github.com/PyTorchLightning/pytorch-lightning/blob/be244560b24b68b0236a4694707fb9bb63c2e6d0/pytorch_lightning/trainer/training_loop.py#L451-L455

Yep, that's the issue 😉

[williamFalcon]

I think I found it #957

[williamFalcon]

Yes... fixed!

@srush make this PR only about the gradnorm?

[williamFalcon]

Check out the video of it working :)

https://twitter.com/PyTorchLightnin/status/1232813118507692033?s=20

[srush]

Awesome! Closing this guy. Switching to #959