From 1db338fdc20f55b862810efb0d00fa71fd873800 Mon Sep 17 00:00:00 2001 From: Merlin Beutlberger Date: Wed, 2 Dec 2020 23:02:24 +0100 Subject: [PATCH 1/3] [INTERNAL] Document the UI5 Tooling benchmarking process PRs where this process was already used: * https://github.com/SAP/ui5-builder/pull/488 * https://github.com/SAP/ui5-builder/pull/390 JIRA: CPOUI5FOUNDATION-261 --- docs/pages/Troubleshooting.md | 149 ++++++++++++++++++++++++++++++++++ 1 file changed, 149 insertions(+) diff --git a/docs/pages/Troubleshooting.md b/docs/pages/Troubleshooting.md index 71298457a8..80ce73fb0d 100644 --- a/docs/pages/Troubleshooting.md +++ b/docs/pages/Troubleshooting.md @@ -8,3 +8,152 @@ You need to delete the HSTS mapping in [chrome://net-internals/#hsts](chrome://n ## Issues Not Listed Here Please follow our [Contribution Guidelines](https://github.com/SAP/ui5-tooling/blob/master/CONTRIBUTING.md#report-an-issue) on how to report an issue. + +## Benchmarking UI5 Tooling + +For benchmarking UI5 Tooling we typically make use of the open source tool [hyperfine](https://github.com/sharkdp/hyperfine). + +In general we only benchmark calls to the UI5 CLI. However, we might add scripted benchmarks for some components in the future. + +The following is a walk-through on how to compare the performance impact of an imaginary change in UI5 Builder project. + +### Setup + +1. Install [hyperfine](https://github.com/sharkdp/hyperfine#installation) +1. Prepare the UI5 Tooling projects you want to measure *(optional if your development environment already reflects this)*: + 1. Start in an empty directory + ```sh + mkdir ui5-tooling-benchmark && cd ui5-tooling-benchmark/ + ``` + 1. Clone [UI5 CLI](https://github.com/SAP/ui5-cli) + ```sh + git clone git@github.com:SAP/ui5-cli.git + ``` + 1. Clone [UI5 Builder](https://github.com/SAP/ui5-builder) (or your fork!) + ```sh + git clone git@github.com:SAP/ui5-builder.git + ``` + Make sure you checkout the `master` branch since we will perform the baseline test first + 1. Install npm dependencies in both directories + ```sh + (cd ui5-cli && npm install) + (cd ui5-builder && npm install) + ``` + 1. Create global npm links for both projects + ```sh + (cd ui5-cli && npm link) + (cd ui5-builder && npm link) + ``` + 1. Link UI5 Builder into UI5 CLI + ```sh + (cd ui5-cli && npm link @ui5/builder) + ``` + 1. Verify your setup + ```sh + ui5 --version + ``` + This should output the version and location of the UI5 CLI you just cloned. + + For example: + ``` + 2.6.6 (from /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js) + ``` + +1. Prepare your test project (we chose the [openui5-sample-app](https://github.com/SAP/openui5-sample-app)) + 1. Clone the project + ```sh + git clone git@github.com:SAP/openui5-sample-app.git + ``` + 1. Navigate into the project + ```sh + cd openui5-sample-app + ``` + 1. Install any required npm dependencies + ```sh + npm install + ``` + Note: We will not link UI5 CLI into the but call the global install directly + 1. Verify that the previously installed UI5 CLI can be called with the following command + ```sh + UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js --version + ``` + On Windows: + ```sh + set UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js --version + ``` + *(Replace the path to ui5.js with the one shown in the previous `ui5 --version` output)* + +### Benchmark + +1. Depending on how reliable you want the measurements, consider preparing your system: + 1. Connect your computer to a power supply + 1. Make sure no updates or anti virus scans are taking place + 1. Close all applications. This includes your IDE since it might start indexing any new files created during the build, thus impacting I/O + 1. Don't interact with your system wile the benchmark is running + +1. Perform the baseline measurement + 1. In the project, start your first benchmark + ```sh + hyperfine --warmup 1 \ + 'UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js build' \ + --export-markdown ./baseline.md + ``` + On Windows: + ```sh + hyperfine --warmup 1 \ + 'set UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js build' \ + --export-markdown ./baseline.md + ``` + 1. Your baseline benchmark is now stored in `baseline.md` and should look similar to this: + + | Command | Mean [s] | Min [s] | Max [s] | Relative | + |:---|---:|---:|---:|---:| + | `UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js build` | 1.439 ± 0.036 | 1.400 | 1.507 | 1.00 | + +1. Prepare your change + 1. Switch to the branch that contains your change + ```sh + (cd ../ui5-builder && git checkout my-change) + ``` + 1. If your change requires different npm dependencies, reinstall them + ```sh + (cd ../ui5-builder && npm install) + ``` + 1. The link from UI5 CLI is still in place. However, if you have changes in **multiple** UI5 Tooling modules, you might need to `npm link` those again + +1. Perform the change measurement + 1. In the project, start your second benchmark + ```sh + hyperfine --warmup 1 \ + 'UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js build' \ + --export-markdown ./my_change.md + ``` + On Windows: + ```sh + hyperfine --warmup 1 \ + 'set UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js build' \ + --export-markdown ./my_change.md + ``` + 1. Your change's benchmark is now stored in `my_change.md` + +### Compile Results + +1. Merge both measurements into one markdown + 1. Either remove the `Relative` column or calculate the relative difference yourself. + 1. You should end up with a markdown like this: + ```md + | Command | Mean [s] | Min [s] | Max [s] | + |:---|---:|---:|---:| + | `UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js build` | 1.439 ± 0.036 | 1.400 | 1.507 | + | `UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js build` | 1.584 ± 0.074 | 1.477 | 1.680 | + ``` + Rendering like this: + + | Command | Mean [s] | Min [s] | Max [s] | + |:---|---:|---:|---:| + | `UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js build` | 1.439 ± 0.036 | 1.400 | 1.507 | + | `UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js build` | 1.584 ± 0.074 | 1.477 | 1.680 | + +1. You can now share these results on GitHub or wherever you might need them! + +**Happy benchmarking! 🏎** From 1cc52aa6b667daa4c37858a4b7e5f697be347b32 Mon Sep 17 00:00:00 2001 From: Merlin Beutlberger Date: Fri, 18 Dec 2020 15:35:08 +0100 Subject: [PATCH 2/3] [INTERNAL] Apply suggestions from code review Co-authored-by: KlattG <57760635+KlattG@users.noreply.github.com> --- docs/pages/Troubleshooting.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/pages/Troubleshooting.md b/docs/pages/Troubleshooting.md index 80ce73fb0d..664a6c9abc 100644 --- a/docs/pages/Troubleshooting.md +++ b/docs/pages/Troubleshooting.md @@ -15,7 +15,7 @@ For benchmarking UI5 Tooling we typically make use of the open source tool [hype In general we only benchmark calls to the UI5 CLI. However, we might add scripted benchmarks for some components in the future. -The following is a walk-through on how to compare the performance impact of an imaginary change in UI5 Builder project. +The following is a walk-through on how to evaluate the performance impact of an imaginary change in UI5 Builder project. ### Setup @@ -33,7 +33,7 @@ The following is a walk-through on how to compare the performance impact of an i ```sh git clone git@github.com:SAP/ui5-builder.git ``` - Make sure you checkout the `master` branch since we will perform the baseline test first + Make sure you check out the `master` branch, since we will perform the baseline test first 1. Install npm dependencies in both directories ```sh (cd ui5-cli && npm install) @@ -59,7 +59,7 @@ The following is a walk-through on how to compare the performance impact of an i 2.6.6 (from /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js) ``` -1. Prepare your test project (we chose the [openui5-sample-app](https://github.com/SAP/openui5-sample-app)) +1. Prepare your test project (we choose the [openui5-sample-app](https://github.com/SAP/openui5-sample-app)) 1. Clone the project ```sh git clone git@github.com:SAP/openui5-sample-app.git @@ -72,7 +72,7 @@ The following is a walk-through on how to compare the performance impact of an i ```sh npm install ``` - Note: We will not link UI5 CLI into the but call the global install directly + Note: We will not link UI5 CLI into this project. Instead, we will call it directly. 1. Verify that the previously installed UI5 CLI can be called with the following command ```sh UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js --version @@ -85,10 +85,10 @@ The following is a walk-through on how to compare the performance impact of an i ### Benchmark -1. Depending on how reliable you want the measurements, consider preparing your system: +1. Depending on how reliable you want the measurements to be, consider preparing your system: 1. Connect your computer to a power supply - 1. Make sure no updates or anti virus scans are taking place - 1. Close all applications. This includes your IDE since it might start indexing any new files created during the build, thus impacting I/O + 1. Make sure no updates or anti-virus scans are taking place + 1. Close all applications. This includes your IDE, since it might start indexing any new files created during the build, thus impacting I/O 1. Don't interact with your system wile the benchmark is running 1. Perform the baseline measurement From 4f31919e653243f964a2212e9b72337051dd73e0 Mon Sep 17 00:00:00 2001 From: Merlin Beutlberger Date: Tue, 19 Jan 2021 15:25:00 +0100 Subject: [PATCH 3/3] [INTERNAL] Apply suggestions from code review Co-authored-by: KlattG <57760635+KlattG@users.noreply.github.com> --- docs/pages/Troubleshooting.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/pages/Troubleshooting.md b/docs/pages/Troubleshooting.md index 664a6c9abc..5d14cb7b2a 100644 --- a/docs/pages/Troubleshooting.md +++ b/docs/pages/Troubleshooting.md @@ -15,7 +15,7 @@ For benchmarking UI5 Tooling we typically make use of the open source tool [hype In general we only benchmark calls to the UI5 CLI. However, we might add scripted benchmarks for some components in the future. -The following is a walk-through on how to evaluate the performance impact of an imaginary change in UI5 Builder project. +The following is a walk-through on how to evaluate the performance impact of an imaginary change in the UI5 Builder project. ### Setup @@ -29,11 +29,11 @@ The following is a walk-through on how to evaluate the performance impact of an ```sh git clone git@github.com:SAP/ui5-cli.git ``` - 1. Clone [UI5 Builder](https://github.com/SAP/ui5-builder) (or your fork!) + 1. Clone [UI5 Builder](https://github.com/SAP/ui5-builder) (or your fork) ```sh git clone git@github.com:SAP/ui5-builder.git ``` - Make sure you check out the `master` branch, since we will perform the baseline test first + Make sure you check out the `master` branch, since we'll perform the baseline test first 1. Install npm dependencies in both directories ```sh (cd ui5-cli && npm install) @@ -72,8 +72,8 @@ The following is a walk-through on how to evaluate the performance impact of an ```sh npm install ``` - Note: We will not link UI5 CLI into this project. Instead, we will call it directly. - 1. Verify that the previously installed UI5 CLI can be called with the following command + Note: We won't link UI5 CLI into this project. Instead, we'll call it directly. + 1. Verify that the previously installed UI5 CLI can be called with the following command: ```sh UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js --version ``` @@ -83,13 +83,13 @@ The following is a walk-through on how to evaluate the performance impact of an ``` *(Replace the path to ui5.js with the one shown in the previous `ui5 --version` output)* -### Benchmark +### Benchmarking -1. Depending on how reliable you want the measurements to be, consider preparing your system: +1. Depending on how reliable you'd like the measurements to be, consider preparing your system: 1. Connect your computer to a power supply 1. Make sure no updates or anti-virus scans are taking place 1. Close all applications. This includes your IDE, since it might start indexing any new files created during the build, thus impacting I/O - 1. Don't interact with your system wile the benchmark is running + 1. Don't interact with your system wile the benchmarking is running 1. Perform the baseline measurement 1. In the project, start your first benchmark @@ -154,6 +154,6 @@ The following is a walk-through on how to evaluate the performance impact of an | `UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js build` | 1.439 ± 0.036 | 1.400 | 1.507 | | `UI5_CLI_NO_LOCAL=X node /my/home/ui5-tooling-benchmark/ui5-cli/bin/ui5.js build` | 1.584 ± 0.074 | 1.477 | 1.680 | -1. You can now share these results on GitHub or wherever you might need them! +1. You can now share these results on GitHub or wherever you might need them. **Happy benchmarking! 🏎**