Using Docker containers from a private registry without a full trust chain (or: use your own DinD container)

[rajbos]

You can use your own runner container rather straightforward, since it is supported in the deployments you can:

• overwrite the spec.image value with your own container label.

# runner.yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: Runner
metadata:
  name: gh-runner
spec:
  repository: robbos/testing-grounds
  image: registry.artifactory.mydomain.com/actions-runner # overwriting the runner to use our own runner image

The DinD runner comes from the controller where you can overwrite it as well. I'll show the yaml below.
I needed to use my own DinD container because our Kubernetes cluster (Rancher in this case) sits behind an internal proxy that sends all requests for images to an internal Artifactory Container Registry. The issue we had came from the Artifactory setup: internal policy dictactes self signed certificates (I know 😱). The bad part is that the internal images don't have a full trust chain setup: that means that a docker pull image for example fails, since the trust chain cannot be verified and docker will reject it. You can try several solutions (more info in my blogpost but in the end this one worked:

1. Create your own DinD container with the certificates backed into the default folder that will be used as 'trusted certificates'
2. Use your own DinD container in the controller deployment of the actions-runner-controller.

1. Create your own DinD container

I created our own DinD container from the default DinD container. Since it is based on Alpine, we need to add our certs to the folder /etc/ssl/certs/. I added all the internal certs that we use, but you can do that with a single cert as well.

# Container definition
FROM docker:dind

# Add the certs from the VM we are running on to this container for secured communication with Artifactory 
COPY /certificates /etc/ssl/certs/

# add the crt to the local certs in the image and call system update on it:
#RUN cat /usr/local/share/ca-certificates/docker_registry.crt >> /etc/ssl/certs/ca-certificates.crt
RUN update-ca-certificates

We where running our GitHub Actions on VM's that where based on Red Hat, so I loaded the certs from the trusted folder there /etc/pki/ca-trust/source/anchors/ and added them to the container:

# GitHub Actions Worklfow commands:
    - uses: actions/checkout@v2
    - name: Build
      run: |
        cd dind
        # certs on RHEL are found here:
        cp -R /etc/pki/ca-trust/source/anchors/ certificates/
        docker build -t ${DIND_NAME}:${TAG} -f Dockerfile --build-arg http_proxy="$http_proxy" --build-arg https_proxy="$http_proxy" --build-arg no_proxy="$no_proxy" .

2. Use your own DinD container in the controller deployment:

To use this container you need to update the controller deployment and include the full spec as below. The changes are in the
label spec.template.spec.containers.args where we start the controller manager with the argument --docker-image=registry.artifactory.mydomain.com/actions-runner-dind:latest:

# controller-manager.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    control-plane: controller-manager
  name: controller-manager
  namespace: actions-runner-system
spec:
  replicas: 1
  selector:
    matchLabels:
      control-plane: controller-manager
  template:
    metadata:
      labels:
        control-plane: controller-manager
    spec:
      containers:
      - args:
        - --metrics-addr=127.0.0.1:8080
        - --enable-leader-election
        - --sync-period=10m
        - --docker-image=registry.artifactory.mydomain.com/actions-runner-dind:latest
        command:
        - /manager
        env:
        - name: GITHUB_TOKEN
          valueFrom:
            secretKeyRef:
              key: github_token
              name: controller-manager
              optional: true
        - name: GITHUB_APP_ID
          valueFrom:
            secretKeyRef:
              key: github_app_id
              name: controller-manager
              optional: true
        - name: GITHUB_APP_INSTALLATION_ID
          valueFrom:
            secretKeyRef:
              key: github_app_installation_id
              name: controller-manager
              optional: true
        - name: GITHUB_APP_PRIVATE_KEY
          value: /etc/actions-runner-controller/github_app_private_key
        image: summerwind/actions-runner-controller:v0.18.2
        name: manager
        ports:
        - containerPort: 9443
          name: webhook-server
          protocol: TCP
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 20Mi
        volumeMounts:
        - mountPath: /tmp/k8s-webhook-server/serving-certs
          name: cert
          readOnly: true
        - mountPath: /etc/actions-runner-controller
          name: controller-manager
          readOnly: true
      - args:
        - --secure-listen-address=0.0.0.0:8443
        - --upstream=http://127.0.0.1:8080/
        - --logtostderr=true
        - --v=10
        image: registry.artifactory.mydomain.com/brancz/kube-rbac-proxy:v0.8.0
        name: kube-rbac-proxy
        ports:
        - containerPort: 8443
          name: https
      terminationGracePeriodSeconds: 10
      volumes:
      - name: cert
        secret:
          defaultMode: 420
          secretName: webhook-server-cert
      - name: controller-manager
        secret:
          secretName: controller-manager

Note: I had to do the same for the label spec.template.spec.containers.manager,image to enable the proxy to find our internal image for kube-rbac-proxy because of something in the Artifactory setup. All other images where correctly mirrored on Artifactory and could be found without additional changes.

[jgrimaud-hum]

I am in a similar situation (behind enterprise firewall)!

Quick question - I assume you needed to define the imagePullSecrets to snag your custom dind sidecar. Does setting the imagePullSecrets attribute of the RunnerDeployment manifest work for pulling in the dind sidecar?

Im getting ready to test this out myself, so if I remember, I will post an update with my findings soon.

[rajbos]

I'm not sure, our container registry isn't protected with a token / secret, only by a firewall. So I didn't need it or could test it. Please let us know! @jgrimaud-hum

[jgrimaud-hum]

It does work for pulling in the dind sidecar. However, I am in a situation where I have to add several custom root and intermediate CAs then route the dind sidecar through internal DNS. Going to be lots of fun... 😆

[rajbos]

Sounds familiar :-). Create your own dind container with the certs, or mount it to the controller-manager yourself.

[jgrimaud-hum]

This worked. Now dealing with an IPv6 loopback on DNS call. Working to define the docker run script for docker:dind. If anyone else thinks this would be helpful info I can post a solution once I get it ironed out. I'm more of a DevOps and process automation engineer and less of a k8s and docker guru...

¯_(ツ)_/¯

[artificial-aidan]

I didn't want to build a docker image with certs embedded in it (🤮 ). So instead I just mount the cert as a secret into the docker image. This should eliminate the need for a custom image. By putting the cert in the directory for the registry, I believe you don't need to install it as a root cert.

apiVersion: actions.summerwind.dev/v1alpha1
kind: Runner
metadata:
  name: ${local.runner_deployment_name}-runner
spec:
    volumes:
    - name: certs
      secret:
        items:
        - key: ca.crt
          path: my-registry:443/ca.crt
        secretName: registry-ca
    dockerVolumeMounts:
    - mountPath: /etc/docker/certs.d
      name: certs
      readOnly: true

[sledigabel]

great to know! thank you

[erichorwath]

Was anybody able to make this work in a Github (enterprise) workflow yaml like this:

- name: Build and push (broker)
  id: docker_build_broker
  uses: docker/build-push-action@v2
  with:
    file: Dockerfile
    push: false #deactivated for testing
    tags: somePrivateRegistryOutsideOfTheCluster/app:test
    cache-from: type=registry,ref=docker-registry.actions-runner-system.svc:5000/app:buildcache
    cache-to: type=registry,ref=docker-registry.actions-runner-system.svc:5000/app:buildcache,mode=max

I'm getting a x509: certificate signed by unknown authority error:

While inside the "docker" container of the Runner pod, I'm able to push images to my cluster internal docker registry with self signed certificates mounted into /etc/docker/certs.d:

Someone any idea?

[rajbos]

Yes. The runner runs in a different container, so you need to mount the certs in that one as well.

[jgrimaud-hum]

Also of importance - the official docker/build-push-action itself creates a container within your dind runner to build and push your workloads. Within that container I also have x509 issues, despite having my enterprise root and CA certs installed on the dind sidecar. I have abandoned using the official docker task for the time being until I have time to research further.

[erichorwath]

@rajbos @jgrimaud-hum thanks for the hint!

Update, buildx has provided a fix: docker/setup-buildx-action#112 (comment)

Now, following is working for me:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: custom-runner-my-org
  namespace: actions-runner-system
spec:
  template:
    spec:
      organization: my-org
      volumes:
      - name: certs
        secret:
          items:
          - key: tls.crt
            path: registry.actions-runner-system.svc:5000/ca.crt
          secretName: registry-tls
      volumeMounts:
      - mountPath: /etc/docker/certs.d
        name: certs
        readOnly: true

cert created via:
(or better use jetstack)

openssl req \
  -newkey rsa:4096 -nodes -sha256 -keyout certs/tls.key \
  -addext "subjectAltName = DNS:registry.actions-runner-system.svc" \
  -x509 -days 365 -out certs/tls.crt
kubectl create secret generic registry-tls -n actions-runner-system \
--from-file=certs/tls.crt --from-file=certs/tls.key

with a local registry installed via:

helm repo add twuni https://helm.twun.io
helm upgrade -i docker-registry twuni/docker-registry \
-n actions-runner-system \
--set tlsSecretName=registry-tls,persistence.enabled=true,fullnameOverride=registry

Those Docker caches are awesome!!!

[jgrimaud-hum]

The last issue I ran into was related to either my enterprise network config or AKS limitations, but I had to modify the /etc/hosts file to enforce localhost as IPv4 to prevent loopbacks.

Found a solution posted by @skyscooby on the moby project #moby/moby/issues/35954

# Insert into entrypoint script

# Inline edits and moves don't work inside containers
sed 's/^::1.*localhost/::1\tip6-localhost/g' /etc/hosts > /etc/hosts.tmp
cat /etc/hosts.tmp > /etc/hosts
rm -f /etc/hosts.tmp

Kind of hacky - but ¯\(ツ)/¯