Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update benchmarks CI with --hard-fail min metric floor #9276

Merged
merged 6 commits into from
Sep 4, 2022

Conversation

glenn-jocher
Copy link
Member

@glenn-jocher glenn-jocher commented Sep 4, 2022

Signed-off-by: Glenn Jocher glenn.jocher@ultralytics.com

πŸ› οΈ PR Summary

Made with ❀️ by Ultralytics Actions

🌟 Summary

Improved benchmark testing with adjustable performance floors.

πŸ“Š Key Changes

  • πŸ”„ Revised the output labels in benchmark results from 'mAP@0.5:0.95' to 'mAP50-95'.
  • ✨ Added a conditional check to apply a minimum performance floor (default 0.29) in benchmarks.
  • πŸ› οΈ Updated --hard-fail CLI argument to accept an optional performance floor value.

🎯 Purpose & Impact

  • πŸ“ Ensures maintainability of performance standards by allowing custom minimum metrics for passing tests.
  • πŸš€ Facilitates detection of performance regressions that could lead to better model quality over time.
  • βš™οΈ Provides more granular control over testing processes, aiding developers in quality assurance tasks.

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
@glenn-jocher
Copy link
Member Author

@AyushExel @kalenmike @sergiossm this is really cool. YOLOv5 CI will now fail if any of the benchmark exports (including base PyTorch model) perform below threshold. I've set the threshold here as 0.29 mAP50-95.

This is the first time I've introduced this capability. In the future we should be able to do this at the model training level also if we get more capable GPU runners.

https://github.com/ultralytics/yolov5/runs/8125360038?check_suite_focus=true
Screenshot 2022-09-04 at 15 56 19

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
@glenn-jocher glenn-jocher changed the title Update benchmarks CI with min metric floor Update benchmarks CI with --hard-fail min metric floor Sep 4, 2022
@glenn-jocher glenn-jocher merged commit e45d335 into master Sep 4, 2022
@glenn-jocher glenn-jocher deleted the update/bench_ci branch September 4, 2022 14:35
ctjanuhowski pushed a commit to ctjanuhowski/yolov5 that referenced this pull request Sep 8, 2022
…#9276)

* Update benchmarks CI with min metric floor

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Update ci-testing.yml

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Update benchmarks.py

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Update benchmarks.py

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Update benchmarks.py

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

* Update benchmarks.py

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant