Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Benchmarking Script #17

Merged
merged 53 commits into from
Feb 9, 2022
Merged

Benchmarking Script #17

merged 53 commits into from
Feb 9, 2022

Conversation

ashwinvaidya17
Copy link
Collaborator

@ashwinvaidya17 ashwinvaidya17 commented Dec 9, 2021 •
edited
Loading

Description

Changes from last PR review

  • Benchmark script uses yaml file for configuration. This file can be used to patch any variable in the existing model config files
  • Move PR template to root
  • Move CDF normalization to functions. The seperate file can copied in the exportable code thus removing duplication on OTE
  • GPU and CPU runs now use different processes as they use separate hardwares

Checklist

  • My code follows the pre-commit style and check guidelines of this project.
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing tests pass locally with my changes

TODO

  • Add benchmarking information to documentation
  • Might need to refactor the files a bit

Relevant Tests

image

image

Ashwin Vaidya added 4 commits December 10, 2021 11:29
Add OpenVINO throughput
855db66
Merge branch 'development' into feature/ashwin/benchmarking_tools
6b78c02
Move pull request template to .github root
9f143ac
Refactor Benchmarking script.
9e8ead8 
Changes
- Normalization support for torch inferencer.
- Move cdf normalization to separate function to reduce future code duplication.
@ashwinvaidya17 ashwinvaidya17 changed the title Initial Benchmarking Script Draft: Benchmarking Script Jan 3, 2022
@ashwinvaidya17 ashwinvaidya17 added the Enhancement New feature or request label Jan 3, 2022
@ashwinvaidya17 ashwinvaidya17 mentioned this pull request Jan 5, 2022
Closed
@samet-akcay samet-akcay linked an issue Jan 5, 2022 that may be closed by this pull request
Inference Bug - Too many variables to unpack #56
Closed
@ashwinvaidya17 ashwinvaidya17 removed a link to an issue Jan 6, 2022
Inference Bug - Too many variables to unpack #56
Closed
@ashwinvaidya17 ashwinvaidya17 changed the base branch from development to feature/refactor January 20, 2022 08:02
@ashwinvaidya17 ashwinvaidya17 marked this pull request as draft January 20, 2022 08:51
Ashwin Vaidya and others added 18 commits February 7, 2022 12:00
Base automatically changed from feature/refactor to development February 8, 2022 15:57
djdameln
djdameln approved these changes Feb 8, 2022
View reviewed changes
Copy link
Contributor

@djdameln djdameln left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments, but overall it looks ready to be merged

tools/benchmarking/benchmark.py Outdated Show resolved Hide resolved
tools/benchmarking/benchmark.py Outdated Show resolved Hide resolved
tools/benchmarking/utils/metrics.py Outdated Show resolved Hide resolved
@ashwinvaidya17
Copy link
Collaborator Author

Looks like the deleted tests are back again. Maybe we can remove those in a different PR.

@samet-akcay
Copy link
Contributor

It's probably because I pulled the development branch. Sorry...

@ashwinvaidya17
Copy link
Collaborator Author

It's probably because I pulled the development branch. Sorry...

That's alright. This was expected a couple of times till all the branches based on refactor get merged.

@samet-akcay samet-akcay merged commit c611e43 into development Feb 9, 2022
@samet-akcay samet-akcay deleted the feature/ashwin/benchmarking_tools branch February 9, 2022 09:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@samet-akcay samet-akcay samet-akcay approved these changes

@djdameln djdameln djdameln approved these changes

Assignees
No one assigned
Labels
Enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Refactor benchmarking script
3 participants
@ashwinvaidya17 @samet-akcay @djdameln