Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "presets" fields to module generation and execution benchmark configs #13683

Closed
pzread opened this issue May 18, 2023 · 0 comments · Fixed by #14204
Closed

Add "presets" fields to module generation and execution benchmark configs #13683

pzread opened this issue May 18, 2023 · 0 comments · Fixed by #14204
Assignees
Labels
infrastructure/benchmark Relating to benchmarking infrastructure infrastructure Relating to build systems, CI, or testing

Comments

@pzread
Copy link
Contributor

pzread commented May 18, 2023

As we start adding large models to the benchmark suites, the single build target iree-benchmark-suites is not fine-grind enough to build what needs to be run and waste space and time to build unrelated large models.

Right now we have benchmark presets to control which ones to run and use tags on benchmark config to map benchmarks to benchmark presets. We can do this more easily by simply having a preset field on the benchmark configs. This also makes benchmark definitions clearly present which presets they belong to.

We can then also split the build targets of benchmark suites according to the presets. Initially we might have the build targets:

  • iree-benchmark-suites-x86_64
  • iree-benchmark-suites-cuda
  • iree-benchmark-suites-vulkan-nvidia
  • iree-benchmark-suites-android-cpu
  • iree-benchmark-suites-android-gpu
  • iree-benchmark-suites-x86_64-large
  • iree-benchmark-suites-cuda-large

And for compilation benchmarks:

  • iree-benchmark-suites-comp-stats
  • iree-benchmark-suites-comp-stats-large

Then the meta build targets:

  • iree-benchmark-suites-default: for default benchmarks
  • iree-benchmark-suites-large: for large benchmarks
@pzread pzread self-assigned this May 18, 2023
@pzread pzread changed the title Add "preset" fields to module generation and execution benchmark configs Add "presets" fields to module generation and execution benchmark configs May 18, 2023
@pzread pzread added infrastructure Relating to build systems, CI, or testing infrastructure/benchmark Relating to benchmarking infrastructure labels May 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure/benchmark Relating to benchmarking infrastructure infrastructure Relating to build systems, CI, or testing
Projects
None yet
1 participant