Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Explicitly set weight_decay value #8592

Merged
merged 3 commits into from
Jul 16, 2022
Merged

Explicitly set weight_decay value #8592

merged 3 commits into from
Jul 16, 2022

Conversation

developer0hye
Copy link
Contributor

@developer0hye developer0hye commented Jul 16, 2022 •
edited by UltralyticsAssistant
Loading

The default weight_decay value of AdamW is 1e-2, so we should set it to zero.

🛠️ PR Summary

Made with ❤️ by Ultralytics Actions

🌟 Summary

Optimization improvements for training in YOLOv5.

📊 Key Changes

  • Set weight_decay to 0.0 for AdamW optimizer when initializing.
  • Explicitly added weight_decay for parameter group g[1] (BatchNorm2d weights), also set to 0.0.

🎯 Purpose & Impact

  • 🎯 Purpose: These changes ensure that weight decay is not applied to BatchNorm2d layers and the bias terms of layers when using the AdamW optimizer, which is common practice for better training stability and performance.
  • 🚀 Impact: Users can expect more stable and potentially improved training results for their machine learning models, particularly when using the AdamW optimizer. This may also help prevent over-regularization of certain parameters in the model.

developer0hye and others added 3 commits July 16, 2022 18:09
@developer0hye
explicitly set weight_decay value
bf39c0f 
The default weight_decay value of AdamW is 1e-2, so we should set it to zero.
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
938559f 
for more information, see https://pre-commit.ci
@glenn-jocher
Cleanup
7779571
@glenn-jocher glenn-jocher changed the title explicitly set weight_decay value Explicitly set weight_decay value Jul 16, 2022
@glenn-jocher
Copy link
Member

@developer0hye wow good catch, I missed the AdamW default decay of 0.01.

@glenn-jocher glenn-jocher merged commit 7204c1c into ultralytics:master Jul 16, 2022
@glenn-jocher
Copy link
Member

@developer0hye PR is merged. Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐

@developer0hye developer0hye deleted the patch-11 branch July 16, 2022 13:55
@developer0hye
Copy link
Contributor Author

@glenn-jocher

Thanks for your reply!

We should re-train yolov5 with this modification!

ctjanuhowski pushed a commit to ctjanuhowski/yolov5 that referenced this pull request Sep 8, 2022
@developer0hye @pre-commit-ci @glenn-jocher
Explicitly set weight_decay value (ultralytics#8592)
7904964 
* explicitly set weight_decay value

The default weight_decay value of AdamW is 1e-2, so we should set it to zero.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Cleanup

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
@Hojland Hojland mentioned this pull request Oct 17, 2022
Merged
@glenn-jocher
Copy link
Member

@developer0hye Thanks for the feedback! If you need any assistance with re-training, feel free to refer to our YOLOv5 documentation for guidance. Happy training!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@developer0hye developer0hye

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@developer0hye @glenn-jocher