Update: YOLO pipeline to return scaled Boxes #881

rahul-tuli · 2023-01-24T16:26:01Z

This PR updates YOLO pipeline to return bounding boxes scaled with respect to the original image rather than preprocessed image.

The solution included two changes:

Maintaining and propagating information on the original image sizes, this was done by adding a return value to the _preprocess function(the original image shape is returned) and propagating it to postprocessing function via the postprocessing_kwargs
Actual rescaling of the bounding boxes based on model input shape and original input shape

code to visualize bounding boxes:

from deepsparse import Pipeline
from deepsparse.utils.annotate import load_image
from deepsparse.yolo import annotate_image
from matplotlib import pyplot as plt


image_path = "./tests/deepsparse/pipelines/sample_images/basilica.jpg"
pipeline = Pipeline.create(task="yolo")
_, original_img = load_image(image_path, image_size=(640, 640))

pipeline_output = pipeline(images=[original_img])
annotated_img = annotate_image(image=original_img, prediction=pipeline_output)
plt.imshow(annotated_img)
plt.show()

Image Before this PR:
Image after this PR:

Note: The failing test, seems to be from the following issue, but I haven't taken a deeper look yet

CONTAINS:

To see the specific tasks where the Asana app for GitHub is being used, see below:
- https://app.asana.com/0/0/1203759380764633

corey-nm

Nice fixes and write up!

src/deepsparse/yolo/pipelines.py

dbogunowicz

LGTM, left some nits

…tion logic to work with both 3 and 4 dimensional inputs

@dbogunowicz

* Update: YOLO pipeline to return scaled Boxes * Apply: comments from @dbogunowicz 's code review * Apply: comments from @dbogunowicz, update original image shape extraction logic to work with both 3 and 4 dimensional inputs * Update: All loaders to only resize an input image if a truthy image_size is passed Update: `annotate_image` signature to not use `model_input_size` as yolo-pipeline take care of scaling now Remove: `--image-shape` argument as it's actually the expected model input shape Which can be inferred from the graph * Update: Yolo Annotation to allow shape overrides (#883) * Update: Annotation script to allow shape override of input onnx model, by adding a `--model_input_shape` CLI argument * Apply: Comments from @bfineran, update `--model-input-shape` to `--model-input-image-shape`

dbogunowicz · 2023-02-01T10:18:50Z

@rahul-tuli shall we land it?

@dbogunowicz

* Update: YOLO pipeline to return scaled Boxes * Apply: comments from @dbogunowicz 's code review * Apply: comments from @dbogunowicz, update original image shape extraction logic to work with both 3 and 4 dimensional inputs * Apply: comments from @dbogunowicz * Update: YOLO Annotation flow to work with rescaled bounding boxes (#882) * Update: YOLO pipeline to return scaled Boxes * Apply: comments from @dbogunowicz 's code review * Apply: comments from @dbogunowicz, update original image shape extraction logic to work with both 3 and 4 dimensional inputs * Update: All loaders to only resize an input image if a truthy image_size is passed Update: `annotate_image` signature to not use `model_input_size` as yolo-pipeline take care of scaling now Remove: `--image-shape` argument as it's actually the expected model input shape Which can be inferred from the graph * Update: Yolo Annotation to allow shape overrides (#883) * Update: Annotation script to allow shape override of input onnx model, by adding a `--model_input_shape` CLI argument * Apply: Comments from @bfineran, update `--model-input-shape` to `--model-input-image-shape` * Fix: Quality Failures

@dbogunowicz

* Update: YOLO pipeline to return scaled Boxes * Apply: comments from @dbogunowicz 's code review * Apply: comments from @dbogunowicz, update original image shape extraction logic to work with both 3 and 4 dimensional inputs * Apply: comments from @dbogunowicz * Update: YOLO Annotation flow to work with rescaled bounding boxes (#882) * Update: YOLO pipeline to return scaled Boxes * Apply: comments from @dbogunowicz 's code review * Apply: comments from @dbogunowicz, update original image shape extraction logic to work with both 3 and 4 dimensional inputs * Update: All loaders to only resize an input image if a truthy image_size is passed Update: `annotate_image` signature to not use `model_input_size` as yolo-pipeline take care of scaling now Remove: `--image-shape` argument as it's actually the expected model input shape Which can be inferred from the graph * Update: Yolo Annotation to allow shape overrides (#883) * Update: Annotation script to allow shape override of input onnx model, by adding a `--model_input_shape` CLI argument * Apply: Comments from @bfineran, update `--model-input-shape` to `--model-input-image-shape` * Fix: Quality Failures Co-authored-by: Rahul Tuli <rahul@neuralmagic.com>

rahul-tuli marked this pull request as ready for review January 24, 2023 16:39

corey-nm previously approved these changes Jan 24, 2023

View reviewed changes

rahul-tuli requested review from bfineran, KSGulin and dbogunowicz January 24, 2023 16:43

rahul-tuli self-assigned this Jan 24, 2023

rahul-tuli added the mle-team label Jan 24, 2023