Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Copy wandb param dict before training to avoid overwrites. #7317

Merged
merged 2 commits into from
Apr 6, 2022
Merged

Copy wandb param dict before training to avoid overwrites. #7317

merged 2 commits into from
Apr 6, 2022

Conversation

n1mmy
Copy link
Contributor

@n1mmy n1mmy commented Apr 6, 2022

Copy the hyperparameter dict retrieved from wandb configuration before passing it to train(). Training overwrites parameters in the dictionary (eg scaling obj/box/cls gains), which causes the values reported in wandb to not match the input values. This is confusing as it makes it hard to reproduce a run, and also throws off wandb's Bayesian sweep algorithm.

🛠️ PR Summary

Made with ❤️ by Ultralytics Actions

🌟 Summary

Improved stability in the Weights & Biases (wandb) sweeping process within YOLOv5 repo.

📊 Key Changes

  • Modified the method of obtaining hyperparameters (hyp_dict) from the wandb sweep agent to include a .copy() operation.

🎯 Purpose & Impact

  • This change ensures that there are no unexpected behaviors during parameter sweeps, as train() modifies parameters that could confuse wandb if the dictionary is not copied.
  • Users conducting hyperparameter sweeps with wandb will experience more reliable and consistent logging and parameter handling. 🛠️

Copy the hyperparameter dict retrieved from wandb configuration before passing it to `train()`. Training overwrites parameters in the dictionary (eg scaling obj/box/cls gains), which causes the values reported in wandb to not match the input values. This is confusing as it makes it hard to reproduce a run, and also throws off wandb's Bayesian sweep algorithm.
@n1mmy
Copy link
Contributor Author

n1mmy commented Apr 6, 2022

Here are screenshots showing the issue. Note the value for cls does not match in the before screenshot, but does match in the after screenshot.
wandb-before
wandb-after

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👋 Hello @n1mmy, thank you for submitting a YOLOv5 🚀 PR! To allow your work to be integrated as seamlessly as possible, we advise you to:

  • ✅ Verify your PR is up-to-date with upstream/master. If your PR is behind upstream/master an automatic GitHub Actions merge may be attempted by writing /rebase in a new comment, or by running the following code, replacing 'feature' with the name of your local branch:
git remote add upstream https://github.com/ultralytics/yolov5.git
git fetch upstream
# git checkout feature  # <--- replace 'feature' with local branch name
git merge upstream/master
git push -u origin -f
  • ✅ Verify all Continuous Integration (CI) checks are passing.
  • ✅ Reduce changes to the absolute minimum required for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." -Bruce Lee

@AyushExel
Copy link
Contributor

Thanks for the PR. Looks good!

@glenn-jocher glenn-jocher merged commit a88a814 into ultralytics:master Apr 6, 2022
@glenn-jocher
Copy link
Member

@n1mmy makes sense, good catch!

PR is merged. Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐

@n1mmy
Copy link
Contributor Author

n1mmy commented Apr 6, 2022

Yay! Thank you both for the quick turnaround and for all your work on this project. I really appreciate it!

BjarneKuehl pushed a commit to fhkiel-mlaip/yolov5 that referenced this pull request Aug 26, 2022
…s#7317)

* Copy wandb param dict before training to avoid overwrites.

Copy the hyperparameter dict retrieved from wandb configuration before passing it to `train()`. Training overwrites parameters in the dictionary (eg scaling obj/box/cls gains), which causes the values reported in wandb to not match the input values. This is confusing as it makes it hard to reproduce a run, and also throws off wandb's Bayesian sweep algorithm.

* Cleanup

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants