Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WandbLogger warning not logging logs. #2015

Closed
rohitgr7 opened this issue May 30, 2020 · 1 comment · Fixed by #2055
Closed

WandbLogger warning not logging logs. #2015

rohitgr7 opened this issue May 30, 2020 · 1 comment · Fixed by #2055
Labels
help wanted Open to be worked on logger Related to the Loggers

Comments

@rohitgr7
Copy link
Contributor

🐛 Bug

WandbLogger giving warning: WARNING Adding to old History rows isn't currently supported. Step 25 < 38 and not logging when I try to use the WandbLogger with k-fold cross-validation because there I am using the same instance of wandb_logger but using trainer.fit multiple times for different train_dl and valid_dl. Since the step gets repeated in each case, it's not logging anything after the 1st fold is complete even though the log keys are completely different. It was working perfectly with pytorch-lightning v-0.7.4. For now, I have to create separate experiments for each fold which are hard to analyze on wandb.

To Reproduce

Code sample

Colab Notebook

Expected behavior

It should log even when the global_step is repeated in case if the logs keys are different.

Environment

  • PyTorch Version (e.g., 1.0): 1.5.0
  • OS (e.g., Linux): Linux
  • How you installed PyTorch (conda, pip, source): pip
  • Build command you used (if compiling from source):
  • Python version: 3.7
  • CUDA/cuDNN version: 10.1
  • GPU models and configuration: Tesla P100-PCIE-16GB
  • Any other relevant information:

Additional context
@rohitgr7 rohitgr7 added the help wanted Open to be worked on label May 30, 2020
@rohitgr7
Copy link
Contributor Author

@borisdayma

borisdayma added a commit to borisdayma/pytorch-lightning that referenced this issue Jun 2, 2020
@borisdayma
fix(wandb): use same logger on multiple training loops
777be31 
New training loops reset step to 0 which would previously try to overwrite logs

fix Lightning-AI#2015
@borisdayma borisdayma mentioned this issue Jun 2, 2020
Merged
5 tasks
williamFalcon pushed a commit that referenced this issue Jun 2, 2020
@borisdayma
fix(wandb): use same logger on multiple training loops (#2055)
00f1ac1 
* fix(wandb): use same logger on multiple training loops

New training loops reset step to 0 which would previously try to overwrite logs

fix #2015

* docs(changelog.md): add reference to PR 2055
justusschock pushed a commit that referenced this issue Jun 29, 2020
@borisdayma @justusschock
fix(wandb): use same logger on multiple training loops (#2055)
9825768 
* fix(wandb): use same logger on multiple training loops

New training loops reset step to 0 which would previously try to overwrite logs

fix #2015

* docs(changelog.md): add reference to PR 2055
@Borda Borda added the logger Related to the Loggers label Aug 4, 2020
This was referenced Oct 15, 2020
Closed
Closed
@borisdayma borisdayma mentioned this issue Oct 28, 2020
Merged
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted Open to be worked on logger Related to the Loggers
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(wandb): use same logger on multiple training loops borisdayma/pytorch-lightning
2 participants
@Borda @rohitgr7