fix dtype/device property not getting updated in submodules

[awaelchli]

What does this PR do?

Fixes #2442

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?
• Did you verify new and existing tests pass locally with your changes?
• If you made a notable change (that affects users), did you update the CHANGELOG?

The new tests fail on master and demonstrate the bug.

[pep8speaks]

Hello @awaelchli! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-07-21 17:32:43 UTC

[codecov]

Codecov Report

Merging #2657 into master will increase coverage by 0%.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master   #2657    +/-   ##
=======================================
  Coverage      91%     91%            
=======================================
  Files          70      72     +2     
  Lines        5778    6131   +353     
=======================================
+ Hits         5270    5599   +329     
- Misses        508     532    +24     

[Borda]

what is the reason of using apply?

[awaelchli]

part of the answer is here in my comment.
apply in contrast to "to" works recursively on all modules and allows us to update our custom properties.
I'm writing a test right now to make sure it fixes what failed before.

[williamFalcon]

@awaelchli can you also test this with dp and ddp_spawn backends?
make sure to use two different functions and not a parametrized test or it won’t work

[justusschock]

Actually I'm kind of clueless why it works now and did not work before, since torch.nn.Module also called apply (e.g. https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/module.py#L462).

But LGTM

[awaelchli]

@justusschock Yes, I understand. It took me a while to figure it out. The thing is, the super calls the _apply, and if you look inside you will see that it will only dispatch the .cuda, .to, etc. to buffers and parameters! But we need our special properties to be set on the module, so we need to call apply (not _apply).