Support returning python scalars in DP

[nsarang]

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?
• If you made a notable change (that affects users), did you update the CHANGELOG?

What does this PR do?

Fixes #1861 and #1904.
I'm not sure if this fix is the right approach. In the docs it's mentioned that training_step must return Tensor (even for the progress_bar dict). But with this PR it'll also work with scalars. Were there other reasons (besides the issues above) to force users to stick with tensors?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

Merging #1935 into master will increase coverage by 1%.
The diff coverage is 90%.

@@          Coverage Diff           @@
##           master   #1935   +/-   ##
======================================
+ Coverage      89%     90%   +1%     
======================================
  Files          79      79           
  Lines        7302    7214   -88     
======================================
- Hits         6514    6496   -18     
+ Misses        788     718   -70     

[williamFalcon]

@nsarang good addition. Can you write a test for this?

[awaelchli]

Could you then also update the docs, since you mention there are wrong informations on the return type?

[Borda]

as mentioned by Will and Adrian, pls

• add tests
• update docs

[Borda]

@nsarang how is it going here? 🐰

[nsarang]

@Borda Sorry I've been busy lately. I'll finish it today 🙂

[pep8speaks]

Hello @nsarang! Thanks for updating this PR.

• In the file pytorch_lightning/core/lightning.py:

Line 190:120: E501 line too long (121 > 119 characters)
Line 362:120: E501 line too long (121 > 119 characters)
Line 804:120: E501 line too long (122 > 119 characters)

Comment last updated at 2020-08-06 12:23:29 UTC

[nsarang]

@Borda
I tested this on 2 GPUs using DP and it was OK. But it gave the following warning:
/nsarang/env-python3.6/lib/python3.6/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.

This warning is supposed to be silenced in ignored_warnings.py but It seems the text doesn't match 🤔

I could add this line to the gather method to avoid this warning.

[mergify]

This pull request is now in conflict... :(

[mergify]

This pull request is now in conflict... :(

[Borda]

@nsarang mind check the tests, pls

[williamFalcon]

mind checking the failing test? @nsarang

[nsarang]

@Borda @williamFalcon
Seems to be working for all tests now except tests/core/test_datamodules.py::test_full_loop_ddp_spawn. The final assertion fails.
I'm not sure how LightningDataParallel is related to ddp_spwan.
Is it because of the way I added alternating batches in TrainingStepVariations and ValidationEpochEndVariations?

[Borda]

@nateraw mind have look? 🐰