Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Square Pad and Resize Preprocessors #2138

Merged
merged 18 commits into from
Jun 13, 2023

Conversation

f4str
Copy link
Collaborator

@f4str f4str commented May 6, 2023

Description

A new module art.preprocessing.image was created which will contain all image preprocessors. Under this module, new image preprocessors for square padding and resizing were created. Both of these preprocessors work for both the classification and object detection tasks. For classification, the labels are not changed. For object detection, the bounding boxes of the labels are shifted/scaled accordingly based on the padding or resizing.

The Square Pad preprocessor (ImageSquarePad, ImageSquarePadPyTorch, and ImageSquarePadTensorFlowV2) will pad all images to be a square shape. The user is capable of specifying the type of padding (e.g., constant or symmetric) and any additional parameters taken by the corresponding framework-specific padding function. This preprocessor can also take either a ndarray/tensor of the same image size or a list of ndarray/tensor of arbitrary sizes and still square pad accordingly.

The Resize preprocessor (ImageResize, ImageResizePyTorch, and ImageResizeTensorFlowV2) will resize all image to the be same user specified height and width. The user is capable of specifying the interpolation method used for resizing (e.g., bilinear or area). This preprocessor can also take either a ndarray/tensor of the same image size or a list of ndarray/tensor of arbitrary sizes and still resize all images to the same dimensions. This preprocessor will always convert the resized images into one large ndarray/tensor so this is ideal last step for the training pipeline.

For training, the Square Pad preprocessor should be followed by the Resize processor so images of arbitrary sizes can be padded to preserve their aspect ratio and then resized to be fit into a model. These preprocessors are necessary for properly training the YOLO estimators.

Fixes #2130
Fixes #2082

Type of change

Please check all relevant options.

  • Improvement (non-breaking)
  • Bug fix (non-breaking)
  • New feature (non-breaking)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

  • Unit tests for Square Pad preprocessor
  • Unit tests for Resize preprocessor

Test Configuration:

  • OS
  • Python version
  • ART version or commit number
  • TensorFlow / Keras / PyTorch / MXNet version

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

f4str added 10 commits April 27, 2023 14:52
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
@codecov-commenter
Copy link

codecov-commenter commented May 6, 2023

Codecov Report

Merging #2138 (7a6eac3) into dev_1.15.0 (43d13fc) will increase coverage by 0.01%.
The diff coverage is 87.78%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Impacted file tree graph

@@              Coverage Diff               @@
##           dev_1.15.0    #2138      +/-   ##
==============================================
+ Coverage       85.64%   85.66%   +0.01%     
==============================================
  Files             299      306       +7     
  Lines           26656    27047     +391     
  Branches         4878     4968      +90     
==============================================
+ Hits            22830    23170     +340     
- Misses           2581     2604      +23     
- Partials         1245     1273      +28     
Impacted Files Coverage Δ
art/preprocessing/image/image_square_pad/numpy.py 86.36% <86.36%> (ø)
...preprocessing/image/image_square_pad/tensorflow.py 86.36% <86.36%> (ø)
...rt/preprocessing/image/image_square_pad/pytorch.py 86.56% <86.56%> (ø)
art/preprocessing/image/image_resize/tensorflow.py 88.52% <88.52%> (ø)
art/preprocessing/image/image_resize/pytorch.py 88.70% <88.70%> (ø)
art/preprocessing/image/image_resize/numpy.py 88.88% <88.88%> (ø)
art/defences/preprocessor/preprocessor.py 78.63% <100.00%> (ø)
art/preprocessing/image/__init__.py 100.00% <100.00%> (ø)

... and 2 files with indirect coverage changes

@f4str f4str marked this pull request as ready for review May 6, 2023 00:43
@beat-buesser beat-buesser self-requested a review May 8, 2023 19:01
@beat-buesser beat-buesser self-assigned this May 8, 2023
@beat-buesser beat-buesser added the enhancement New feature or request label May 8, 2023
@beat-buesser beat-buesser added this to the ART 1.15.0 milestone May 8, 2023
Copy link
Collaborator

@beat-buesser beat-buesser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @f4str Thank you very much for your pull request! I have added a few questions, what do you think?

art/preprocessing/image/image_square_pad/tensorflow.py Outdated Show resolved Hide resolved
tests/preprocessing/image/test_image_resize.py Outdated Show resolved Hide resolved
tests/preprocessing/image/test_image_square_pad.py Outdated Show resolved Hide resolved
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
@f4str
Copy link
Collaborator Author

f4str commented May 11, 2023

Hi @beat-buesser thank you for the comments! I've addressed the review comments and have made the corresponding changes. Feel free to let me know if there any other issues.

@f4str f4str requested a review from beat-buesser May 11, 2023 05:16
Copy link
Collaborator

@beat-buesser beat-buesser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @f4str Thank you very much for contributing image preprocessors to ART!

@beat-buesser beat-buesser merged commit f1b6a50 into Trusted-AI:dev_1.15.0 Jun 13, 2023
@f4str f4str deleted the object-detector-resize branch June 13, 2023 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
3 participants