Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multi arch image builds #45

Merged
merged 2 commits into from
Jul 5, 2021
Merged

Multi arch image builds #45

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
39ff519
GHA for building and pushing images
RealHarshThakur Jun 21, 2021
2f61d91
Add date to image tag
RealHarshThakur Jun 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions .github/workflows/image.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
name: release

on:
create:
push:
branches:
- master
RealHarshThakur marked this conversation as resolved.
Show resolved Hide resolved
- 'v*'
tags:
- 'v*'

# TODO: Migrate building binaries from Cloud Build
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/binaries/manifests and kubectl plugins/

jobs:
hnc-manager:
RealHarshThakur marked this conversation as resolved.
Show resolved Hide resolved
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v1

- name: Set Release Tag
run: |
TAG="${GITHUB_REF#refs/*/v}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this give you if you're running on master?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It gives it as refs/heads/master whereas when it's pushed to v1, it gives it as 1.

https://github.com/RealHarshThakur/hierarchical-namespaces/runs/2753272153?check_suite_focus=true#step:8:11
https://github.com/RealHarshThakur/hierarchical-namespaces/runs/2753141639?check_suite_focus=true

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This means that we'll repeatedly overwrite the same tag, right? I'm not a huge fan of that, especially on the branches. Can we append the commit hash, e.g. something like hnc_manager:master-0a1b2c-multiarch for the master branch, and hnc_manager:0.9-3d4e5f-multiarch for the branches?

For the tags, it's probably ok to just have the tag name, e.g. hnc_manager:0.9.0-multiarch. We'll never build the same tag twice - if we need to do a release candidate, we'll include that in the tag name in Git (e.g. 0.9.0-rc1) and if we need to make a change to an existing tag, we'll just create a new one instead (e.g. 0.9.1).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could also not push image for master branch. Only when a release is made and a branch is created, we create an image for it. We can have a separate image tag for Github Actions, just hope it doesn't confuse the end users. WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This applies to the release branches as well. It's only Git tags that only get built once.

I'd prefer if we could append some unique prefix onto each build that comes from a branch (and not a tag). Failing that, can we say -latest on it so people know not to rely on it being constant?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I suggest doing a +{date} to the tag as these are automatic builds. A 'real' release should likely only be retagging an intermediate build into the proper version. At least that would be my suggestion, especially since our release include more than just an image build. It also includes docs, kustomization files, and soon to be helm charts.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RealHarshThakur I'm not seeing the date in the tab above, am I missing someething?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rjbez17 added it here

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that sets the Docker tag so that it includes the date, but RELEASE_TAG (which I'm guessing is the Github release tag) still includes only the branch name, correct? E.g., I'd prefer for this to be something like:

GITREF="${GITHUB_REF#refs/*/v}"
echo "RELEASE_TAG=${GITREF}-${{ steps.date.outputs.HNC_BUILD_TIMESTAMP }}" >> $GITHUB_ENV

In other words, line up the RELEASE_TAG and the Docker tags as closely as possible.

@RealHarshThakur if I've misinterpreted please feel free to remove the hold. We can always clean it up later anyway.

echo "RELEASE_TAG=${TAG}" >> $GITHUB_ENV

- name: Set Build Date
id: date
run: |
echo "::set-output name=HNC_BUILD_TIMESTAMP::$(date -u +'%Y-%m-%dT%H:%M:%S%Z')"

- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@master
with:
# GCR_KEY is the service account key in json which is base64 encoded and stored in the secrets section of the repo.
service_account_key: ${{ secrets.GCR_KEY }}
RealHarshThakur marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you say anything about what this service account needs to be able to do? E.g., it must have permission to push to gcr.io/k8s-staging-multitenancy? If that's the requirement, how did you test this - by pushing to some different GCR repo?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used the Default compute service account as shown in this blog and yes, I tested with a different repo.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. That's bad advice, the default compute service account is terrible and generally shouldn't be used :D But there was no way for you to know that! I take it that the account was in the same GCP project as the repo you used to test?

Maybe just add the comment that we think the requirement for the service accounts it that it can push to the required registry (sorry, "registry" not "repo") but that we're not certain. We'll find out soon enough!

export_default_credentials: true

# Configure docker to use the gcloud command-line tool as a credential helper
- run: |
# Set up docker to authenticate
# via gcloud command-line tool.
gcloud auth configure-docker

- name: Docker meta
id: docker_meta
uses: docker/metadata-action@v3
with:
images: gcr.io/k8s-staging-multitenancy/hnc-manager
tags: |
type=raw,value=multiarch-{{branch}}

- name: Print Tags
run: |
echo "${{ steps.docker_meta.outputs.tags }}"
adrianludwin marked this conversation as resolved.
Show resolved Hide resolved
echo "RELEASE TAG: ${RELEASE_TAG}"

## QEMU is an emulator which makes it possible to transalte instruction sets to different platforms.
## In this step, we setup the driver which Docker can use to build images on different platforms
- name: Set up QEMU
adrianludwin marked this conversation as resolved.
Show resolved Hide resolved
uses: docker/setup-qemu-action@v1
with:
platforms: all

- name: Set up Docker Buildx
id: buildx
uses: docker/setup-buildx-action@v1
with:
version: v0.5.1


- name: Build & Push Image
uses: docker/build-push-action@v2
with:
context: .
file: ./Dockerfile
push: true
platforms: linux/amd64, linux/arm64, linux/arm/v7
tags: |
${{ steps.docker_meta.outputs.tags }}
build-args: |
DBUILD_DATE=${{ steps.date.outputs.HNC_BUILD_TIMESTAMP }}
DBUILD_REPO_URL=https://github.com/kubernetes-sigs/hierarchical-namespaces
2 changes: 1 addition & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ COPY cmd/ cmd/
COPY internal/ internal/

# Build
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager ./cmd/manager/main.go
RUN CGO_ENABLED=0 GO111MODULE=on go build -a -o manager ./cmd/manager/main.go
adrianludwin marked this conversation as resolved.
Show resolved Hide resolved

# Copied from kubebuilder scaffold to run as nonroot at
# https://github.com/kubernetes-sigs/kubebuilder/blob/7af89cb00c224c57ece37dc14ea37caf1eb769db/pkg/scaffold/v2/dockerfile.go#L60
Expand Down
24 changes: 24 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,14 @@ build: generate fmt vet manifests
-o bin/kubectl/kubectl-hns_darwin_amd64 \
-ldflags="-X sigs.k8s.io/hierarchical-namespaces/internal/version.Version=${HNC_IMG_TAG}" \
./cmd/kubectl/main.go
GOOS=linux GOARCH=arm64 go build \
-o bin/kubectl/kubectl-hns_linux_arm64 \
-ldflags="-X sigs.k8s.io/hierarchical-namespaces/internal/version.Version=${HNC_IMG_TAG}" \
./cmd/kubectl/main.go
GOOS=linux GOARCH=arm go build \
-o bin/kubectl/kubectl-hns_linux_arm \
-ldflags="-X sigs.k8s.io/hierarchical-namespaces/internal/version.Version=${HNC_IMG_TAG}" \
./cmd/kubectl/main.go

# Clean all binaries (manager and kubectl)
clean: krew-uninstall
Expand Down Expand Up @@ -232,6 +240,22 @@ docker-build: generate fmt vet
@echo "Warning: this does not run tests. Run 'make test' to ensure tests are passing."
docker build . -t ${HNC_IMG}


buildx-setup:
## This script needs to be run to start the emulator for multiarch builds
# This driver translates the instruction set to different platforms
docker run --rm --privileged linuxkit/binfmt:v0.8
docker buildx create --use --name=qemu
docker buildx inspect --bootstrap


# Build and push multi-arch image
RealHarshThakur marked this conversation as resolved.
Show resolved Hide resolved
docker-push-multi: buildx-setup generate fmt vet
@echo "Warning: this does not run tests. Run 'make test' to ensure tests are passing."
docker buildx build \
--push \
--platform linux/arm64,linux/amd64,linux/arm/v7 --tag ${HNC_IMG} .

###################### KIND ACTIONS #########################

# Creates a local kind cluster, destroying the old one if necessary.
Expand Down
2 changes: 1 addition & 1 deletion config/default/manager_auth_proxy_patch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ spec:
spec:
containers:
- name: kube-rbac-proxy
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.4.0
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0
args:
# The value of --upstream must match the value of --metrics-addr passed
# to the manager binary in the base config in
Expand Down