Expose OpenVINO batch_size similarly to TensorRT

[democat3457]

This PR exposes OpenVINO's network batch_size similarly to TensorRT's batch_size in DetectMultiBackend. This allows for a more consistent and cross-backend indication of whether a fixed non-one batch size exists and what the value of the batch size is.

Use case:
Current way to get a model's fixed batch size

model = DetectMultiBackend(path, device=torch.device(device))
batch_size = model.batch_size if hasattr(model, "batch_size") else None
if model.xml: # explicit handling of openvino
  batch_size = model.network.batch_size

New way to get a model's fixed batch size

model = DetectMultiBackend(path, device=torch.device(device))
batch_size = model.batch_size if hasattr(model, "batch_size") else None
# no need for explicit handling of openvino, code is much cleaner

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

🌟 Summary

Enhanced OpenVINO integration with dynamic batch size support for the YOLOv5 model.

📊 Key Changes

• Added a line to fetch and store the batch size from the OpenVINO network object.

🎯 Purpose & Impact

• Purpose: This change aims to allow dynamic handling of batch sizes within the YOLOv5 model when using OpenVINO as the inference backend. Instead of assuming a fixed batch size, the model can now operate with the batch size that is provided by the loaded network.
• Impact: This improvement enables greater flexibility for YOLOv5 users that deploy their models using OpenVINO, potentially optimizing performance for varying batch sizes during inference.🔧 It can lead to more efficient use of computational resources and better support for different deployment scenarios.

[github-actions]

👋 Hello @democat3457, thank you for submitting a YOLOv5 🚀 PR! To allow your work to be integrated as seamlessly as possible, we advise you to:

• ✅ Verify your PR is up-to-date with upstream/master. If your PR is behind upstream/master an automatic GitHub Actions merge may be attempted by writing /rebase in a new comment, or by running the following code, replacing 'feature' with the name of your local branch:

git remote add upstream https://github.com/ultralytics/yolov5.git
git fetch upstream
# git checkout feature  # <--- replace 'feature' with local branch name
git merge upstream/master
git push -u origin -f

• ✅ Verify all Continuous Integration (CI) checks are passing.
• ✅ Reduce changes to the absolute minimum required for your bug fix or feature addition. "It is not daily increase but daily decrease, hack away the unessential. The closer to the source, the less wastage there is." -Bruce Lee

[glenn-jocher]

@democat3457 yes good idea! PR is merged. Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐

[glenn-jocher]

@democat3457 looks like I didn't investigate the CI enough. Even though benchmarks CI passed the OpenVINO inference failed due to network not having a batch_size property. A user discovered this in #8509

I'll revert the PR to give you time to investigate and see if there's a better way to handle this. To reproduce:

!python export.py --include openvino
!python detect.py --weights yolov5s_openvino_model/

[democat3457]

Hmm interesting, I'll look into it

[glenn-jocher]

@democat3457 great thanks!

[democat3457]

@glenn-jocher I can't reproduce using the above steps, using openvino-dev==2022.1, Ubuntu 20.04

$ python detect.py --weights yolov5s_openvino_model/
detect: weights=['yolov5s_openvino_model/'], source=data/images, data=data/coco128.yaml, imgsz=[640, 640], conf_thres=0.25, iou_thres=0.45, max_det=1000, device=, view_img=False, save_txt=False, save_conf=False, save_crop=False, nosave=False, classes=None, agnostic_nms=False, augment=False, visualize=False, update=False, project=runs/detect, name=exp, exist_ok=False, line_thickness=3, hide_labels=False, hide_conf=False, half=False, dnn=False
YOLOv5 🚀 v6.1-283-g39d7a93 Python-3.8.10 torch-1.11.0+cu102 CUDA:0 (NVIDIA GeForce GTX 1080 Ti, 11176MiB)

Loading yolov5s_openvino_model for OpenVINO inference...
image 1/2 /home/user/yolov5/data/images/bus.jpg: 640x640 4 persons, 1 bus, Done. (0.171s)
image 2/2 /home/user/yolov5/data/images/zidane.jpg: 640x640 2 persons, 2 ties, Done. (0.144s)
Speed: 1815.6ms pre-process, 157.5ms inference, 4.5ms NMS per image at shape (1, 3, 640, 640)
Results saved to runs/detect/exp4

trying to use openvino-dev==2021.4.2 gives an error on running detect.py:

Loading yolov5s_openvino_model for OpenVINO inference...
Traceback (most recent call last):
  File "detect.py", line 256, in <module>
    main(opt)
  File "detect.py", line 251, in main
    run(**vars(opt))
  File "/home/user/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "detect.py", line 92, in run
    model = DetectMultiBackend(weights, device=device, dnn=dnn, data=data, fp16=half)
  File "/home/user/yolov5/models/common.py", line 364, in __init__
    from openvino.runtime import Core
ModuleNotFoundError: No module named 'openvino.runtime'

[glenn-jocher]

@democat3457 oh yes it's currently fixed in master (v6.1-283) because I reverted this PR in #8510

If you want to test this PR #8437 (yours) then you can checkout the commit before I reverted, and you should see "v6.1-282"
i.e. this produces the error.

cd yolov5
git checkout dd28df98c2307abfe13f8857110bfcd6b5c4eb4b
python detect.py --weights yolov5s_openvino_model/

[democat3457]

oh I see why, I didn't pull the newest changes to the DetectMultiBackend before testing this. oops

(also the commit is 27d831b, not dd28df9)

[glenn-jocher]

@democat3457 got it. Please submit a new PR if you see a way to re-implement correctly. I'm working on better benchmarking CI that should fail on errors to make sure we don't let this happen again on the CI side in #8513, should be done soon.