GPU Usage Logger

[groadabike]

What does this PR do?

This is a Callback that logs the GPU utilisation during the training stage, giving a sense of how the resources have been used, helping to measure the effect of any implementation improvement.

Callback that query NVIDIA-SMI some GPU stats and log it:

• "fan.speed"
• "memory.used"
• "memory.free"
• "utilization.memory"
• "utilization.cpu
• "temperature.gpu"
• "temperature.memory"

Also, It measures the time between batches (inter_step_time) and in batches (intra_step_time)

These last two parameters can give a sense of how the "total batch time" is distributed (loading batch + batch pass).
For example, a large inter_step_time could mean a slow dataload and any improvement in the DataSet class it will be reflected in a reduction of this parameter.

Fixes # (2074)

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you to create a separate PR for every change.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?
• Did you verify new and existing tests pass locally with your changes?
• If you made a notable change (that affects users), did you update the CHANGELOG?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[pep8speaks]

Hello @groadabike! Thanks for updating this PR.

• In the file pytorch_lightning/callbacks/gpu_usage_logger.py:

Line 92:49: E127 continuation line over-indented for visual indent
Line 115:49: E127 continuation line over-indented for visual indent
Line 141:108: W292 no newline at end of file

Comment last updated at 2020-08-12 14:17:24 UTC

[williamFalcon]

how is this different than the flag to log gpu usage?

[groadabike]

how is this different than the flag to log gpu usage?

the log_gpu_memory flag in trainer just log the memory.used or the min and max memory used.
It doesn't log the rest of the parameters included in the PR.
for example, the % of gpu utilisation or memory free.

[codecov]

Codecov Report

Merging #2932 into master will increase coverage by 3%.
The diff coverage is 40%.

@@           Coverage Diff           @@
##           master   #2932    +/-   ##
=======================================
+ Coverage      86%     90%    +3%     
=======================================
  Files          80      81     +1     
  Lines        7449    7542    +93     
=======================================
+ Hits         6430    6755   +325     
+ Misses       1019     787   -232     

[williamFalcon]

ok cool. makes sense!

let's clean up the docs for this