diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 51c6ac53a8e9..ff62038cce25 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1089,3 +1089,51 @@ jobs: with: webhook: ${{ secrets.DISCORD_WEBHOOK }} description: "The following jobs failed: ${{ steps.failed_jobs.outputs.failed-jobs }}" + - name: Show useful artifact links + if: always() + env: + BUILD_DIR_GCS_ARTIFACT: ${{ needs.build_all.outputs.build-dir-gcs-artifact }} + E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-gcs-artifact-dir }} + BENCHMARK_TOOLS_GCS_ARTIFACT_DIR: ${{ needs.build_benchmark_tools.outputs.benchmark-tools-gcs-artifact-dir }} + EXECUTION_BENCHMARK_RESULTS_GCS_ARTIFACT_DIR: ${{ needs.execution_benchmarks.outputs.benchmark-results-gcs-artifact-dir }} + COMPILATION_BENCHMARK_RESULTS_GCS_ARTIFACT: ${{ needs.compilation_benchmarks.outputs.compile-stats-results-gcs-artifact }} + # If the job of an artifact is skipped or failed, we show "Not present". + run: | + cat <> "${GITHUB_STEP_SUMMARY}" + ### :link: Artifact Links + + \`\`\`sh + # Compiler and host tools archive: + export BUILD_DIR_URL="${BUILD_DIR_GCS_ARTIFACT:-Not present}" + # Directory of e2e test artifacts: + export E2E_TEST_ARTIFACTS_DIR_URL="${E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR:-Not present}" + # Directory of benchmark tool binaries: + export BENCHMARK_TOOLS_DIR_URL="${BENCHMARK_TOOLS_GCS_ARTIFACT_DIR:-Not present}" + # Directory of execution benchmark results and traces: + export EXECUTION_BENCHMARK_RESULTS_DIR_URL="${EXECUTION_BENCHMARK_RESULTS_GCS_ARTIFACT_DIR:-Not present}" + # Compilation benchmark results: + export COMPILATION_BENCHMARK_RESULTS_URL="${COMPILATION_BENCHMARK_RESULTS_GCS_ARTIFACT:-Not present}" + \`\`\` + + You can list \`gs://\` directories and download files with [gcloud cli]( + https://cloud.google.com/sdk/gcloud/reference/storage). + +
+ Click to show common usages + + \`\`\`sh + # Download all artifacts + mkdir /tmp/iree_e2e_test_artifacts + gcloud storage cp -r "\${E2E_TEST_ARTIFACTS_DIR_URL?}/*" /tmp/iree_e2e_test_artifacts + \`\`\` + + \`\`\`sh + # Download execution benchmark results and traces + mkdir /tmp/iree_benchmark_results + gcloud storage cp -r "\${EXECUTION_BENCHMARK_RESULTS_DIR_URL?}/*" /tmp/iree_benchmark_results + \`\`\` +
+ + To run benchmarks locally with the CI-built e2e test artifacts, see [IREE Benchmark Suites]( + https://github.com/${GITHUB_REPOSITORY}/blob/main/docs/developers/developing_iree/benchmark_suites.md#3-fetch-the-benchmark-artifacts). + EOF diff --git a/docs/developers/developing_iree/benchmark_suites.md b/docs/developers/developing_iree/benchmark_suites.md index b17721b9b0aa..465eddb9ad93 100644 --- a/docs/developers/developing_iree/benchmark_suites.md +++ b/docs/developers/developing_iree/benchmark_suites.md @@ -152,12 +152,12 @@ If you don't have artifacts locally, see find the GCS directory of the CI artifacts. Then fetch the needed files: ```sh -export GCS_URL="gs://iree-github-actions--artifacts//" +# Get ${E2E_TEST_ARTIFACTS_DIR_URL} from "Fetching Benchmark Artifacts from CI". export E2E_TEST_ARTIFACTS_DIR="e2e_test_artifacts" # Download all artifacts mkdir "${E2E_TEST_ARTIFACTS_DIR?}" -gcloud storage cp -r "${GCS_URL?}/e2e-test-artifacts" "${E2E_TEST_ARTIFACTS_DIR?}" +gcloud storage cp -r "${E2E_TEST_ARTIFACTS_DIR_URL?}" "${E2E_TEST_ARTIFACTS_DIR?}" ``` Run the helper tool to dump benchmark commands from benchmark configs: @@ -188,22 +188,18 @@ build_tools/benchmarks/benchmark_helper.py dump-cmds \ #### 1. Find the corresponding CI workflow run On the commit of the benchmark run, you can find the list of the workflow jobs -by clicking the green check mark. Click the `Details` of job -`build_e2e_test_artifacts`: +by clicking the green check mark. Click any job starts with `CI /`: -![image](https://user-images.githubusercontent.com/2104162/223781032-c22e2922-2bd7-422d-abc2-d6ef0d31b0f8.png) +![image](https://user-images.githubusercontent.com/2104162/234647960-3df9d0f0-a34a-47ad-bda8-095ae44de865.png) -#### 2. Get the GCS directory of the built artifacts +#### 2. Get URLs of GCS artifacts -On the job detail page, expand the step `Uploading e2 test artifacts`, you will -see a bunch of lines like below. The -URL`gs://iree-github-actions--artifacts///` -is the directory of artifacts: +On the CI page, click `Summary` on the top-left to open the summary page. Scroll +down and the links to artifacts are listed in a section titled "Artifact Links". +Paste the content in your shell to define all needed variables for the following +steps: -``` -Copying file://build-e2e-test-artifacts/e2e_test_artifacts/iree_MobileBertSquad_fp32_module_fdff4caa105318036534bd28b76a6fe34e6e2412752c1a000f50fafe7f01ef07/module.vmfb to gs://iree-github-actions-postsubmit-artifacts/4360950546/1/e2e-test-artifacts/iree_MobileBertSquad_fp32_module_fdff4caa105318036534bd28b76a6fe34e6e2412752c1a000f50fafe7f01ef07/module.vmfb -... -``` +![image](https://user-images.githubusercontent.com/2104162/234716421-3a69b6ad-211d-4e39-8f9e-a4f22f91739d.png) #### 3. Fetch the benchmark artifacts @@ -214,21 +210,17 @@ more usages). If you want to use CI artifacts to reproduce benchmarks locally, see [Find Compile and Run Commands to Reproduce Benchmarks](#find-compile-and-run-commands-to-reproduce-benchmarks). -Set the GCS directory URL from the step -[2. Get the GCS directory of the built artifacts](#2-get-the-gcs-directory-of-the-built-artifacts): - -```sh -export GCS_URL="gs://iree-github-actions--artifacts//" -``` +Assume you get the GCS URL variables from +[Get URLs of GCS artifacts](#2-get-urls-of-gcs-artifacts). Download artifacts: ```sh # The GCS directory has the same structure as your local ${IREE_BUILD_DIR?}/e2e_test_artifacts. -gcloud storage ls "${GCS_URL?}/e2e-test-artifacts" +gcloud storage ls "${E2E_TEST_ARTIFACTS_DIR_URL?}" # Download all source and imported MLIR files: -gcloud storage cp "${GCS_URL?}/e2e-test-artifacts/*.mlir" "" +gcloud storage cp "${E2E_TEST_ARTIFACTS_DIR_URL?}/*.mlir" "" ``` Execution and compilation benchmark configs can be downloaded at: @@ -236,21 +228,24 @@ Execution and compilation benchmark configs can be downloaded at: ```sh # Execution benchmark config: gcloud storage cp \ - "${GCS_URL?}/e2e-test-artifacts/execution-benchmark-config.json" \ + "${E2E_TEST_ARTIFACTS_DIR_URL?}/execution-benchmark-config.json" \ "${E2E_TEST_ARTIFACTS_DIR?}/exec_config.json" # Compilation benchmark config: gcloud storage cp \ - "${GCS_URL?}/e2e-test-artifacts/compilation-benchmark-config.json" \ + "${E2E_TEST_ARTIFACTS_DIR_URL?}/compilation-benchmark-config.json" \ "${E2E_TEST_ARTIFACTS_DIR?}/comp_config.json" ``` Benchmark raw results and traces can be downloaded at: ```sh -# Benchmark raw results -gcloud storage cp "${GCS_URL?}/benchmark-results/benchmark-results-*.json" . +# Execution benchmark raw results +gcloud storage cp "${EXECUTION_BENCHMARK_RESULTS_DIR_URL?}/benchmark-results-*.json" . + +# Execution benchmark traces +gcloud storage cp "${EXECUTION_BENCHMARK_RESULTS_DIR_URL?}/benchmark-traces-*.tar.gz" . -# Benchmark traces -gcloud storage cp "${GCS_URL?}/benchmark-results/benchmark-traces-*.tar.gz" . +# Compilation benchmark results +gcloud storage cp "${COMPILATION_BENCHMARK_RESULTS_URL?}" . ```