Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add full encryption/decryption functionalities for image datasets to DatumaroBinary format #831

Merged

Conversation

vinnamkim
Copy link
Contributor

@vinnamkim vinnamkim commented Feb 28, 2023

Summary

  • Ticket no. 104650
  • Finish to support full encryption/decryption functionalities for image datasets to DatumaroBinary format.
    • Move Crypter to datumaro/components
    • Save images and media with encryption if Crypter is given.
  • Because of performance issues, refactor annotation mappers to use backward_dict().

How to test

I added unit tests and an integration test for this feature.

Checklist

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below)
# Copyright (C) 2023 Intel Corporation
#
# SPDX-License-Identifier: MIT

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
 - Refactor CommonSemanticSegmentation unit tests as well

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
 - Support DatasetItem by Datumaro binary format but annotations are not yet.

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
…maro-binary-format-dataset-item

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@vinnamkim vinnamkim changed the title Support full encryption/decryption functionalities for image datasets. Support full encryption/decryption functionalities for image datasets Feb 28, 2023
@vinnamkim vinnamkim changed the title Support full encryption/decryption functionalities for image datasets Add full encryption/decryption functionalities for image datasets to DatumaroBinary format Feb 28, 2023
@vinnamkim vinnamkim added ENHANCE Enhancement of existing features data formats PR is related to dataset formats labels Mar 2, 2023
 - Import yolo -> export DatumaroBinary (encryption) -> import
DatumaroBinary (encryption) -> Export yolo

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@vinnamkim vinnamkim added this to the 1.1.0 milestone Mar 6, 2023
@vinnamkim vinnamkim force-pushed the feature/encrypt-datumaro-binary branch 6 times, most recently from 4fc561e to 95dcb4b Compare March 8, 2023 12:59
…datumaro-binary

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@vinnamkim vinnamkim force-pushed the feature/encrypt-datumaro-binary branch from 95dcb4b to 8d23d28 Compare March 8, 2023 13:13
@vinnamkim vinnamkim marked this pull request as ready for review March 8, 2023 13:18
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@codecov-commenter
Copy link

codecov-commenter commented Mar 8, 2023

Codecov Report

Patch coverage: 86.69% and project coverage change: -0.04 ⚠️

Comparison is base (6a739e3) 78.53% compared to head (12f144a) 78.49%.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #831      +/-   ##
===========================================
- Coverage    78.53%   78.49%   -0.04%     
===========================================
  Files          205      206       +1     
  Lines        24832    24939     +107     
  Branches      5004     5007       +3     
===========================================
+ Hits         19502    19577      +75     
- Misses        4208     4236      +28     
- Partials      1122     1126       +4     
Flag Coverage Δ
macos-11_Python-3.8 77.48% <86.69%> (-0.04%) ⬇️
ubuntu-20.04_Python-3.8 78.49% <86.69%> (-0.04%) ⬇️
windows-2019_Python-3.8 78.43% <86.69%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...o/plugins/data_formats/datumaro_binary/importer.py 85.00% <40.00%> (-15.00%) ⬇️
datumaro/util/image.py 88.32% <75.00%> (-2.08%) ⬇️
datumaro/components/crypter.py 80.85% <80.85%> (ø)
datumaro/components/media.py 83.46% <85.18%> (-0.10%) ⬇️
...umaro/plugins/data_formats/datumaro_binary/base.py 93.15% <90.00%> (+0.19%) ⬆️
...o/plugins/data_formats/datumaro_binary/exporter.py 97.87% <93.10%> (-2.13%) ⬇️
.../data_formats/datumaro_binary/mapper/annotation.py 98.10% <94.20%> (-1.90%) ⬇️
datumaro/components/exporter.py 84.32% <100.00%> (+4.62%) ⬆️
datumaro/plugins/data_formats/datumaro/exporter.py 95.65% <100.00%> (+0.03%) ⬆️
...aro/plugins/data_formats/datumaro_binary/format.py 88.88% <100.00%> (+0.65%) ⬆️

... and 2 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Contributor

@wonjuleee wonjuleee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please hold this until reviewing cryptography-related security tasks.

wonjuleee
wonjuleee previously approved these changes Mar 17, 2023
Copy link
Contributor

@wonjuleee wonjuleee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The codes look great to me.
But, let's do below in the future

  • separate an option for media encryption
  • reduce the encryption/decryption overhead
  • provide jupyter notebook or cli tutorial for the practical usage of data encryption

…datumaro-binary

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@vinnamkim
Copy link
Contributor Author

vinnamkim commented Mar 17, 2023

The codes look great to me. But, let's do below in the future

I made tickets to resolve your concerns:

  • separate an option for media encryption
  • Ticket no. 106290
  • reduce the encryption/decryption overhead
  • Ticket no. 105335
  • provide jupyter notebook or cli tutorial for the practical usage of data encryption
  • Ticket no. 105333

@vinnamkim vinnamkim merged commit e601f3e into openvinotoolkit:develop Mar 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data formats PR is related to dataset formats ENHANCE Enhancement of existing features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants