Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add python benchmarks.py --test for export-only #7350

Merged
merged 2 commits into from
Apr 8, 2022

Conversation

glenn-jocher
Copy link
Member

@glenn-jocher glenn-jocher commented Apr 8, 2022

Export tests. CPU example. @kalenmike

Colab Pro+ CPU

$ python utils/benchmarks.py --weights yolov5n.pt --test --device cpu

benchmarks: weights=yolov5n.pt, imgsz=640, batch_size=1, data=/content/yolov5/data/coco128.yaml, device=cpu, half=False, test=True
Checking setup...
YOLOv5 πŸš€ v6.1-117-g7e47852 torch 1.10.0+cu111 CPU
Setup complete βœ… (8 CPUs, 51.0 GB RAM, 42.5/166.8 GB disk)

Exports complete (219.55s)
                   Format  Export
0                 PyTorch    True
1             TorchScript    True
2                    ONNX    True
3                OpenVINO    True
4                TensorRT   False
5                  CoreML    True
6   TensorFlow SavedModel    True
7     TensorFlow GraphDef    True
8         TensorFlow Lite    True
9     TensorFlow Edge TPU    True
10          TensorFlow.js    True

Colab Pro+ GPU

$ python utils/benchmarks.py --weights yolov5n.pt --test --device 0

benchmarks: weights=yolov5n.pt, imgsz=640, batch_size=1, data=/content/yolov5/data/coco128.yaml, device=0, half=False, test=True
Checking setup...
YOLOv5 πŸš€ v6.1-117-g7e47852 torch 1.10.0+cu111 CUDA:0 (Tesla V100-SXM2-16GB, 16160MiB)
Setup complete βœ… (8 CPUs, 51.0 GB RAM, 47.2/166.8 GB disk)

Exports complete (636.24s)
                   Format  Export
0                 PyTorch    True
1             TorchScript    True
2                    ONNX    True
3                OpenVINO    True
4                TensorRT    True
5                  CoreML   False
6   TensorFlow SavedModel    True
7     TensorFlow GraphDef    True
8         TensorFlow Lite    True
9     TensorFlow Edge TPU    True
10          TensorFlow.js    True

πŸ› οΈ PR Summary

Made with ❀️ by Ultralytics Actions

🌟 Summary

Improvements and new features added to the benchmarking and exporting capabilities of YOLOv5 models.

πŸ“Š Key Changes

  • Added a new test flag to the benchmarking script to test model exports exclusively.
  • Refined assertions for better handling and support checking for Edge TPU and TF.js export formats.
  • Introduced a new function test() which tests only the exports, providing faster feedback on export validity.
  • Enhanced logging with clear information whether the tests were on benchmarks or just exports.
  • Both benchmark and export tests now have structured output in DataFrame form for post-process analysis and logging.

🎯 Purpose & Impact

  • πŸŽ› The test flag and its associated test() function allow developers to quickly verify the export process without running full benchmarks, increasing efficiency.
  • πŸ”§ Better assertions and structured error messages offer improved developer experience by quickly identifying non-supported export formats for certain devices.
  • ⏱ Enhanced DataFrame outputs help in better tracking and understanding the performance and success of model exports.
  • πŸš€ Overall, this PR aims to streamline the development workflow, reducing the time needed to test and verify model export functionality, and enabling sharper focus on model deployment readiness.

@glenn-jocher glenn-jocher self-assigned this Apr 8, 2022
@glenn-jocher glenn-jocher merged commit 698a5d7 into master Apr 8, 2022
@glenn-jocher glenn-jocher deleted the update/export_tests branch April 8, 2022 23:32
BjarneKuehl pushed a commit to fhkiel-mlaip/yolov5 that referenced this pull request Aug 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant