Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix flushing loggers #1459

Merged
merged 17 commits into from
Apr 15, 2020
Merged

fix flushing loggers #1459

merged 17 commits into from
Apr 15, 2020

Conversation

Borda
Copy link
Member

@Borda Borda commented Apr 11, 2020

Before submitting

  • Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?
  • If you made a notable change (that affects users), did you update the CHANGELOG?

What does this PR do?

Fixes #1447 and #1460.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

@Borda Borda added the bug Something isn't working label Apr 11, 2020
@Borda Borda added this to the 0.7.4 milestone Apr 11, 2020
@mergify mergify bot requested review from a team April 11, 2020 21:39
@Borda Borda marked this pull request as ready for review April 11, 2020 22:29
@Borda
Copy link
Member Author

Borda commented Apr 12, 2020

I will add a tests to cover this usecases over all loggers....

@codecov
Copy link

codecov bot commented Apr 12, 2020

Codecov Report

Merging #1459 into master will increase coverage by 0%.
The diff coverage is 100%.

@@          Coverage Diff           @@
##           master   #1459   +/-   ##
======================================
  Coverage      91%     91%           
======================================
  Files          67      67           
  Lines        3742    3756   +14     
======================================
+ Hits         3400    3415   +15     
+ Misses        342     341    -1     

@pep8speaks
Copy link

pep8speaks commented Apr 12, 2020

Hello @Borda! Thanks for updating this PR.

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-04-14 17:19:52 UTC

@Borda
Copy link
Member Author

Borda commented Apr 12, 2020

I have added parametrized test for "fit + test" and "pickle" for all loggers which are running with default

@Borda
Copy link
Member Author

Borda commented Apr 12, 2020

any idea why unrelated test benchmark fails:

        # verify torch.backends.cudnn.benchmark is not turned on
>       assert not torch.backends.cudnn.benchmark
E       AssertionError: assert not True
E        +  where True = <module 'torch.backends.cudnn' from '/home/circleci/.local/lib/python3.6/site-packages/torch/backends/cudnn/__init__.py'>.benchmark
E        +    where <module 'torch.backends.cudnn' from '/home/circleci/.local/lib/python3.6/site-packages/torch/backends/cudnn/__init__.py'> = <module 'torch.backends' from '/home/circleci/.local/lib/python3.6/site-packages/torch/backends/__init__.py'>.cudnn
E        +      where <module 'torch.backends' from '/home/circleci/.local/lib/python3.6/site-packages/torch/backends/__init__.py'> = torch.backends

@Borda Borda added priority: 0 High priority task ci Continuous Integration labels Apr 13, 2020
@rank_zero_only
def finalize(self, status: str = None) -> None:
# super().finalize(status)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this intended to be commented? If so, why?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added this to be the same as other loggers but then it is crashing because the accumulator is missing...
I don't feel like debugging this magic tool, there are already several other things I don't like much...
It have not been even properly tested yet

@mergify mergify bot requested a review from a team April 13, 2020 14:20
@Borda Borda requested a review from neggert April 14, 2020 00:00
@Borda Borda changed the title fix flushing loggers fix flushing loggers [wip] Apr 14, 2020
@Borda
Copy link
Member Author

Borda commented Apr 14, 2020

opened issue here - https://github.community/t5/GitHub-Actions/Hanging-workflow-even-with-successful-tests/m-p/53850#M8988

UPDATE: it was the __del__ in logger which cased the hanging...

@Borda Borda changed the title fix flushing loggers [wip] fix flushing loggers Apr 14, 2020
@Borda Borda added the ready PRs ready to be merged label Apr 14, 2020
@williamFalcon williamFalcon merged commit b3fe17d into master Apr 15, 2020
@Borda Borda deleted the bugfix/flush-logger branch April 15, 2020 06:23
@pitercl
Copy link
Contributor

pitercl commented Apr 15, 2020

Hi @Borda, it seems that you turned offline_mode on in NeptuneLogger by default. Was this intentional?

If not, how would you like to proceed with restoring the previous default - should I throw a PR or would you like to do it?

@williamFalcon
Copy link
Contributor

@Borda let’s turn the default back on

tullie pushed a commit to tullie/pytorch-lightning that referenced this pull request Jun 7, 2020
* flushing loggers

* flushing loggers

* flushing loggers

* flushing loggers

* changelog

* typo

* fix trains

* optimize imports

* add logger test all

* add logger test pickle

* flake8

* fix benchmark

* hanging loggers

* try

* del

* all

* cleaning
@Borda Borda modified the milestones: 0.7.4, v0.7.x Apr 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working ci Continuous Integration priority: 0 High priority task ready PRs ready to be merged
Projects
None yet
5 participants