Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run a k8s rolling restart of the deployment in CRIB after develop image is built #12532

Merged
merged 4 commits into from
Mar 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions .github/workflows/build-publish-develop.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,49 @@ jobs:
dockerhub_username: ${{ secrets.DOCKERHUB_READONLY_USERNAME }}
dockerhub_password: ${{ secrets.DOCKERHUB_READONLY_PASSWORD }}
git-commit-sha: ${{ steps.git-ref.outputs.checked-out || github.sha }}

- name: Setup GAP
# Don't run for plugins.
if: matrix.image.name == ''
uses: smartcontractkit/.github/actions/setup-gap@1bc7ce34fa81fffcb4a6eb0e4e12e59d94d0fc8f # setup-gap@0.2.0
with:
aws-region: ${{ secrets.AWS_REGION }}
aws-role-arn: ${{ secrets.AWS_OIDC_IAM_ROLE_ARN }}
api-gateway-host: ${{ secrets.AWS_API_GW_HOST_K8S_SAND }}
use-k8s: "true"
k8s-cluster-name: ${{ secrets.AWS_EKS_CLUSTER_NAME_SAND }}
use-private-ecr-registry: true
ecr-private-registry: ${{ secrets.AWS_ACCOUNT_ID_PROD }}
metrics-job-name: push-chainlink-develop ${{ matrix.image.name }}
gc-host: ${{ secrets.GRAFANA_INTERNAL_HOST }}
gc-basic-auth: ${{ secrets.GRAFANA_INTERNAL_BASIC_AUTH }}
gc-org-id: ${{ secrets.GRAFANA_INTERNAL_TENANT_ID }}

# A mutable image tag is used for these CRIBs and it was just built/published
# from this workflow. The deployment has an `imagePullPolicy: Always` set, so
# we need to restart the deployments to pick up the new image.
- name: Restart K8s Deployments for CRIBs
# Don't run for plugins.
if: matrix.image.name == ''
shell: bash
run: |
set -euo pipefail
# Removes the "smartcontractkit/" (org name) prefix.
REPO_NAME_ONLY="${GITHUB_REPOSITORY##*/}"
K8S_NAMESPACE="crib-${REPO_NAME_ONLY}-develop"

deployment_node_names=$(kubectl --namespace "${K8S_NAMESPACE}" \
get deployments \
-l "app=${K8S_NAMESPACE}" \
-o custom-columns=:metadata.name --no-headers)

IFS=$'\n' read -r -d '' -a deployment_names_arr <<< "$deployment_node_names" || :
for name in "${deployment_names_arr[@]}"; do
echo "Restarting deployment: $name"
kubectl --namespace "${K8S_NAMESPACE}" \
rollout restart "deployment/${name}"
done

- name: Collect Metrics
if: always()
id: collect-gha-metrics
Expand Down
45 changes: 0 additions & 45 deletions .github/workflows/build-publish-pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,51 +53,6 @@ jobs:
dockerhub_username: ${{ secrets.DOCKERHUB_READONLY_USERNAME }}
dockerhub_password: ${{ secrets.DOCKERHUB_READONLY_PASSWORD }}

- name: Get PR labels
id: pr-labels
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.number }}
run: |
RESPONSE=$(gh pr view ${PR_NUMBER} --json labels)
# Check if the labels command was successful
if [[ $? -ne 0 ]]; then
echo "Error fetching labels"
exit 1
fi
echo "RESPONSE=${RESPONSE}"
LABELS=$(echo "$RESPONSE" | jq -r '.labels | map(.name) | join(", ")')
# Check if any labels were found
if [[ -z "${LABELS:-}" ]]; then
echo "No labels found"
else
echo "labels=${LABELS}" | tee -a "${GITHUB_OUTPUT}"
fi

- name: Setup GAP
if: contains(steps.pr-labels.outputs.labels, 'crib')
uses: smartcontractkit/.github/actions/setup-gap@main
with:
aws-region: ${{ secrets.AWS_REGION }}
aws-role-arn: ${{ secrets.AWS_OIDC_IAM_ROLE_PUBLISH_PR_ARN }}
api-gateway-host: ${{ secrets.AWS_API_GW_HOST_ARGO_SAND }}
use-argocd: "true"
argocd-user: ${{ secrets.ARGOCD_USER_SAND }}
argocd-pass: ${{ secrets.ARGOCD_PASS_SAND }}

# Run an Argo CD sync after the image is built.
- name: Argo CD App Sync
if: contains(steps.pr-labels.outputs.labels, 'crib')
shell: bash
env:
PR_NUMBER: ${{ github.event.number }}
run: |
argocd app sync \
--plaintext \
--grpc-web \
--async \
"crib-chainlink-${PR_NUMBER}"

- name: Collect Metrics
if: always()
id: collect-gha-metrics
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ kind: Deployment
{{ end }}
metadata:
name: {{ $.Release.Name }}-{{ $cfg.name }}-db
labels:
app: {{ $.Release.Name }}-db
instance: {{ $cfg.name }}-db
release: {{ $.Release.Name }}
spec:
{{ if $.Values.db.stateful }}
serviceName: {{ $.Release.Name }}-db-${{ $cfg.name }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,13 @@ apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ if eq $index 0 }}{{ $.Release.Name }}-{{ $cfg.name }}-bootstrap{{ else }}{{ $.Release.Name }}-{{ $cfg.name }}{{ end }}
labels:
app: {{ $.Release.Name }}
instance: {{ $cfg.name }}
release: {{ $.Release.Name }}
{{- range $key, $value := $.Values.labels }}
{{ $key }}: {{ $value | quote }}
{{- end }}
spec:
strategy:
# Need to recreate the pod to deal with lease lock held by old pod.
Expand Down
4 changes: 4 additions & 0 deletions charts/chainlink-cluster/templates/geth-deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ apiVersion: apps/v1
kind: Deployment
metadata:
name: geth-{{ $cfg.networkId }}
labels:
app: geth
release: {{ $.Release.Name }}
instance: geth-{{ $cfg.networkId }}
spec:
selector:
matchLabels:
Expand Down
Loading