-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ci: buildkite release pipeline #509
Open
v1v
wants to merge
49
commits into
main
Choose a base branch
from
feature/configure-macos-run
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
49 commits
Select commit
Hold shift + click to select a range
ee08db9
Create catalog-info file
v1v d863062
rename and avoid running except explitly
v1v 604506f
bk: pipeline steps doing nothing
v1v 61098a3
buildkite: run package on MacOS without signing
v1v 1960bb1
sign linux gpg and skip notarize
v1v c48f8d2
Merge remote-tracking branch 'upstream/main' into feature/configure-m…
v1v 0fadb4b
merge
v1v edd2c3d
use shared secret
v1v a80da7d
use vault kv for shared secrets
v1v bcead0e
fix slack
v1v c20546b
log group in buildkite
v1v 873375e
trying to support nvm
v1v 0023a8d
avoid with the upload
v1v ec94d95
try something else
v1v b40f664
debug
v1v faa331c
avoid errors
v1v eb1b9bd
something else
v1v b5c6a7c
[skip-ci] source zshrc
v1v 522992e
[skip ci] avoid failing
v1v 1a119c3
fix: configure token to download artifacts
v1v cef9922
fix: download artifacts
v1v 8101e16
Sign windows
v1v b244a4c
upload
v1v ae756a2
fix precommand
v1v c22f0cc
macos
v1v 0a35510
prepare next steps
v1v 9298478
use ext
v1v eac7047
refactor
v1v 7d269bd
fix extra space
v1v 0280ccb
use main
v1v 3de93d8
Merge branch 'feature/configure-macos-run' of https://github.com/elas…
v1v a47c2a0
Update .buildkite/release.yml
v1v 4827e2f
fix double quotes
v1v 36ac844
chore: enable s3 bucket
v1v a894fd0
no need to use a different protocol, it uses buidkite
v1v 5863845
create folder
v1v 2817d3d
support for ephemeral github token
v1v db2de99
support download signed artifacts only
v1v 575816f
chore: for testing purposes
v1v bc7c437
docs
v1v d0a4c1b
fix
v1v 2a1d1b8
fix
v1v 637ff68
supprot no dry-run
v1v 71d6f82
support tag releases
v1v 3400b4c
support dry-run dynamically
v1v 4ac9085
Revert "support dry-run dynamically"
v1v 444c53a
support dry-run if no tag
v1v e28f2ff
use signed-artifacts to be able to publish them
v1v 3dc92c8
test it works the github secret token
v1v File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
#!/usr/bin/env bash | ||
## This script prepares the secret context | ||
## | ||
## NOTE: *_SECRET or *_TOKEN env variables are masked, hence if you'd like to avoid any | ||
## surprises please use the suffix _SECRET or _TOKEN for those values that contain | ||
## any sensitive data. Buildkite can mask those values automatically | ||
|
||
set -eo pipefail | ||
|
||
# To help with testing the GPG signing | ||
BUILDKITE_TOKEN_SECRET=$(vault kv get -field=buildkite_token kv/ci-shared/observability-ci/buildkite-read-build-access) | ||
export BUILDKITE_TOKEN_SECRET | ||
|
||
# If not Git tag release then DRY_RUN=true | ||
DRY_RUN=true | ||
if [ -n "${BUILDKITE_TAG}" ] ; then | ||
DRY_RUN=false | ||
fi | ||
export DRY_RUN | ||
|
||
# Upload should only allow configuring the Buildkite Token ih the pre-command. | ||
if [[ "$BUILDKITE_COMMAND" =~ .*"upload".* ]]; then | ||
echo "Skipped pre-command when running the Upload pipeline" | ||
# NOTE: exit 0 with the pre-command does not work! | ||
else | ||
echo "~~~ Configure ephemeral GitHub token" | ||
GITHUB_TOKEN=$VAULT_GITHUB_TOKEN | ||
GH_TOKEN=$VAULT_GITHUB_TOKEN | ||
export GH_TOKEN GITHUB_TOKEN | ||
set +x | ||
set +e | ||
echo "~~~ Install nvm" | ||
touch ~/.zshrc # See https://github.com/nvm-sh/nvm?tab=readme-ov-file#troubleshooting-on-macos | ||
curl -o- https://github.com/raw/nvm-sh/nvm/v0.39.7/install.sh | bash | ||
fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -2,40 +2,70 @@ | |||||
# $yaml-language-server: $schema=https://github.com/raw/buildkite/pipeline-schema/main/schema.json | ||||||
steps: | ||||||
- label: "Package :package:" | ||||||
command: "true" | ||||||
command: ".buildkite/scripts/package.sh" | ||||||
agents: | ||||||
provider: orka | ||||||
imagePrefix: generic-13-ventura-x64 | ||||||
|
||||||
- wait: ~ | ||||||
|
||||||
- label: "Sign linux :linux:" | ||||||
command: "true" | ||||||
- label: GPG Sign artifacts | ||||||
trigger: unified-release-gpg-signing | ||||||
# NOTE: If you change 'key: gpg-sign-service' then change the bellow call for .buildkite/scripts/download-signed-artifacts.sh | ||||||
key: gpg-sign-service | ||||||
build: | ||||||
env: | ||||||
INPUT_PATH: buildkite:// | ||||||
|
||||||
- label: "Sign macos :mac:" | ||||||
command: "true" | ||||||
- label: MacOS Sign artifacts | ||||||
trigger: unified-release-macos-signing | ||||||
# NOTE: If you change 'key: macos-sign-service' then change the bellow call for .buildkite/scripts/download-signed-artifacts.sh | ||||||
key: macos-sign-service | ||||||
build: | ||||||
env: | ||||||
INPUT_PATH: buildkite:// | ||||||
|
||||||
- label: "Sign windows :windows:" | ||||||
command: "true" | ||||||
- label: Windows Sign artifacts | ||||||
trigger: unified-release-windows-signing | ||||||
# NOTE: If you change 'key: windows-sign-service' then change the bellow call for .buildkite/scripts/download-signed-artifacts.sh | ||||||
key: windows-sign-service | ||||||
build: | ||||||
env: | ||||||
DOWNLOAD_ARTIFACTS_FILTER: "*.exe" | ||||||
INPUT_PATH: buildkite:// | ||||||
|
||||||
- wait: ~ | ||||||
|
||||||
- label: Download signed artifacts | ||||||
commands: | ||||||
- .buildkite/scripts/download-signed-artifacts.sh "gpg-sign-service" "gpg" | ||||||
- .buildkite/scripts/download-signed-artifacts.sh "macos-sign-service" "macos" | ||||||
- .buildkite/scripts/download-signed-artifacts.sh "windows-sign-service" "windows" | ||||||
- .buildkite/scripts/download-signed-artifacts.sh "gpg-sign-service" "gpg" | buildkite-agent pipeline upload | ||||||
- .buildkite/scripts/download-signed-artifacts.sh "macos-sign-service" "macos" | buildkite-agent pipeline upload | ||||||
- .buildkite/scripts/download-signed-artifacts.sh "windows-sign-service" "windows" | buildkite-agent pipeline upload | ||||||
|
||||||
- wait: ~ | ||||||
|
||||||
- label: "Publish S3 Artifacts :s3:" | ||||||
command: "true" | ||||||
trigger: unified-release-publish-s3-artifacts | ||||||
key: publish-s3-service | ||||||
build: | ||||||
env: | ||||||
DESTINATION_PATH: "s3://download.elasticsearch.org/synthetics-recorder/" | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. need to exclude the subfolders but use the files in the root directory. Waitign for the RElease team to provide this feature |
||||||
DRY_RUN: ${DRY_RUN} | ||||||
|
||||||
- wait: ~ | ||||||
|
||||||
- label: "Publish GitHub Release :github:" | ||||||
command: "true" | ||||||
commands: | ||||||
- .buildkite/scripts/create-github-release.sh | ||||||
depends_on: | ||||||
- "gpg" | ||||||
- "macos" | ||||||
- "windows" | ||||||
|
||||||
# Figure out how to notify releases (maybe subscribe the slack channel to the GitHub releases?) | ||||||
notify: | ||||||
- slack: "#observablt-bots" | ||||||
- slack: "#on-week-oblt-productivity" | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What channel?
Suggested change
|
||||||
if: 'build.state != "passed"' | ||||||
|
||||||
# Review the message format for noting when it failed or not. | ||||||
notify: | ||||||
- slack: | ||||||
channels: | ||||||
- "#on-week-oblt-productivity" | ||||||
message: | | ||||||
Release done. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
#!/usr/bin/env bash | ||
# Download the signed artifacts and run the GitHub release using the GH cli | ||
# | ||
# Required environment variables: | ||
# - BUILDKITE_TAG | ||
# | ||
set -eox pipefail | ||
|
||
DIST_LOCATION=signed-artifacts | ||
echo "--- Download signed artifacts" | ||
buildkite-agent artifact download --step gpg "$DIST_LOCATION/*.*" ./ | ||
buildkite-agent artifact download --step macos "$DIST_LOCATION/*.*" ./ | ||
buildkite-agent artifact download --step windows "$DIST_LOCATION/*.*" ./ | ||
|
||
echo "--- List signed artifacts" | ||
ls -l "$DIST_LOCATION/" | ||
|
||
echo "--- Install gh :github:" | ||
if ! gh --version &>/dev/null ; then | ||
wget -q https://github.com/cli/cli/releases/download/v2.50.0/gh_2.50.0_linux_amd64.tar.gz -O gh.tar.gz | ||
tar -xpf gh.tar.gz --strip-components=2 | ||
PATH="$(pwd):${PATH}" | ||
export PATH | ||
gh --version | ||
fi | ||
|
||
echo "--- Run GitHub release" | ||
if [ -n "${BUILDKITE_TAG}" ] ; then | ||
# VAULT_GITHUB_TOKEN is the GitHub ephemeral token created in Buildkite | ||
GH_TOKEN=$VAULT_GITHUB_TOKEN \ | ||
gh release \ | ||
create \ | ||
"${BUILDKITE_TAG}" \ | ||
--draft \ | ||
--title "${BUILDKITE_TAG}" \ | ||
--repo elastic/synthetics-recorder \ | ||
"${DIST_LOCATION}/*.*" | ||
else | ||
echo "gh release won't be triggered this is not a Git tag release, but let's list the releases" | ||
# VAULT_GITHUB_TOKEN is the GitHub ephemeral token created in Buildkite | ||
GH_TOKEN=$VAULT_GITHUB_TOKEN \ | ||
gh release list --repo elastic/synthetics-recorder | ||
fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
#!/usr/bin/env bash | ||
# | ||
# Create a dynamic buildkite step with the artifacts to be downloaded | ||
# | ||
# Required environment variables: | ||
# - BUILDKITE_TOKEN_SECRET | ||
# | ||
|
||
STEP=$1 | ||
DOWNLOAD_STEP_NAME=$2 | ||
|
||
## Support main pipeline and downstream pipelines | ||
if [ -n "$BUILDKITE_TRIGGERED_FROM_BUILD_PIPELINE_SLUG" ] ; then | ||
BUILDKITE_PIPELINE_SLUG=$BUILDKITE_TRIGGERED_FROM_BUILD_PIPELINE_SLUG | ||
BUILDKITE_BUILD_NUMBER=$BUILDKITE_TRIGGERED_FROM_BUILD_NUMBER | ||
fi | ||
|
||
## Fail if no token | ||
if [ -z "$BUILDKITE_TOKEN_SECRET" ] ; then | ||
echo "Token could not be loaded from vault. Please review .buildkite/hooks/pre-command" | ||
exit 1 | ||
fi | ||
|
||
BUILDS_URL="https://api.buildkite.com/v2/organizations/elastic/pipelines/$BUILDKITE_PIPELINE_SLUG/builds" | ||
build_json=$(curl -sH "Authorization: Bearer $BUILDKITE_TOKEN_SECRET" "$BUILDS_URL/$BUILDKITE_BUILD_NUMBER") | ||
# sign-service is the pipeline step in .buildkite/release.yml | ||
SIGN_BUILD_ID=$(jq -r ".jobs[] | select(.step_key == \"$STEP\").triggered_build.id" <<< "$build_json") | ||
|
||
## Fail if no build id | ||
if [ -z "$SIGN_BUILD_ID" ] ; then | ||
echo "Sign build id could not be found. Please review $BUILDS_URL/$BUILDKITE_BUILD_NUMBER and the below json output:" | ||
echo "$build_json" | ||
curl -sH "Authorization: Bearer $BUILDKITE_TOKEN_SECRET" "$BUILDS_URL" | ||
exit 1 | ||
fi | ||
|
||
cat << EOF | ||
- label: ":pipeline: Download signed artifacts" | ||
key: "$DOWNLOAD_STEP_NAME" | ||
commands: | ||
- mkdir -p signed-artifacts | ||
- buildkite-agent artifact download --build "$SIGN_BUILD_ID" "*.*" signed-artifacts/ | ||
- buildkite-agent artifact upload "signed-artifacts/*.*" | ||
agents: | ||
image: docker.elastic.co/ci-agent-images/ubuntu-build-essential | ||
EOF |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
#!/usr/bin/env bash | ||
set -exo pipefail | ||
|
||
echo "~~~ Load nvm" | ||
if [ -n "$BUILDKITE" ] ; then | ||
set +xe | ||
# Need to figure out what's the reason NVM is not explictly loaded | ||
source "$HOME/.zshrc" | ||
fi | ||
|
||
echo "--- Install node and gather dependencies" | ||
nvm install "$(cat .nvmrc)" | ||
npm ci | ||
|
||
echo "--- run release-ci" | ||
# Disable signing | ||
# see https://www.electron.build/code-signing#how-to-disable-code-signing-during-the-build-process-on-macos | ||
export CSC_IDENTITY_AUTO_DISCOVERY=false | ||
# Disable notarize, see scripts/notarize.js | ||
export SKIP_NOTARIZATION=true | ||
npm run release-ci | ||
|
||
# Store unsigned artifacts | ||
if [ -n "$BUILDKITE" ] ; then | ||
echo "--- Upload artifacts" | ||
mv dist artifacts-to-sign | ||
# (only *nix) | ||
buildkite-agent artifact upload "artifacts-to-sign/*.deb;artifacts-to-sign/*.dmg;artifacts-to-sign/*.zip" | ||
|
||
# (only windows) | ||
buildkite-agent artifact upload "artifacts-to-sign/*.exe" | ||
|
||
# (only mac) | ||
buildkite-agent artifact upload "artifacts-to-sign/*.dmg" | ||
fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is an ongoing task to move away from BK API Tokens and use some opinionated contracts so we can skip this API call for the sake of simplicity. Release team is working on it