[FEATURE] Explicit support for running k3s in docker mode.

[rpocase]

Problem

I have been experimenting with k3d as a lightweight method for CI and development workflows. My target deployment environment ultimately has a hard requirement on k3s to be running with --docker due to lack of support for other container run times.

I can start a k3d cluster (with or without worker nodes), and the cluster goes 'running' and check cluster-info shows services, but the k3s server fails to completely launch any pods.

k3d create -n dev -x '--docker' --volume '/var/run/docker.sock:/var/run/docker.sock'          2019/10/01 07:50:04 Created cluster network with ID 159cd009c71407c9fbf7607a7d9b1513a11bcd6d5fcefd1c478f8462930ed0bb
2019/10/01 07:50:04 Created docker volume  k3d-dev-images
2019/10/01 07:50:04 Creating cluster [dev]
2019/10/01 07:50:04 Creating server using docker.io/rancher/k3s:v0.7.0...
2019/10/01 07:50:05 SUCCESS: created cluster [dev]
2019/10/01 07:50:05 You can now use the cluster with:

export KUBECONFIG="$(k3d get-kubeconfig --name='dev')"
kubectl cluster-info

export KUBECONFIG="$(k3d get-kubeconfig --name='dev')"
kubectl cluster-info
Kubernetes master is running at https://localhost:6443
CoreDNS is running at https://localhost:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Digging into the server logs, I see a lot of similar error messages.

W1001 12:51:30.951241       1 docker_sandbox.go:386] failed to read pod IP from plugin/docker: NetworkPlugin cni failed on the 
status hook for pod "coredns-b7464766c-kvjml_kube-system": Unexpected command output nsenter: can't open '/proc/26668/ns/net': 
No such file or directory
 with error: exit status 1
...
E1001 12:51:31.923002       1 kuberuntime_manager.go:697] createPodSandbox for pod "coredns-b7464766c-kvjml_kube-system(09ed1b31-e44a-11e9-9ee8-0242c0a87002)" failed: rpc error: code = Unknown desc = rewrite resolv.conf failed for pod "coredns-b7464766c-kvjml": ResolvConfPath "/var/lib/docker/containers/677c106291d8a8052504be9f451d8fe4c8960c603f7f1339f9939e4976beca1e/resolv.conf" does not exist
E1001 12:51:31.923187       1 pod_workers.go:190] Error syncing pod 09ed1b31-e44a-11e9-9ee8-0242c0a87002 ("coredns-b7464766c-kvjml_kube-system(09ed1b31-e44a-11e9-9ee8-0242c0a87002)"), skipping: failed to "CreatePodSandbox" for "coredns-b7464766c-kvjml_kube-system(09ed1b31-e44a-11e9-9ee8-0242c0a87002)" with CreatePodSandboxError: "CreatePodSandbox for pod \"coredns-b7464766c-kvjml_kube-system(09ed1b31-e44a-11e9-9ee8-0242c0a87002)\" failed: rpc error: code = Unknown desc = rewrite resolv.conf failed for pod \"coredns-b7464766c-kvjml\": ResolvConfPath \"/var/lib/docker/containers/677c106291d8a8052504be9f451d8fe4c8960c603f7f1339f9939e4976beca1e/resolv.conf\" does not exist"
E1001 12:51:31.983323       1 remote_runtime.go:109] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = rewrite resolv.conf failed for pod "helm-install-traefik-hbldm": ResolvConfPath "/var/lib/docker/containers/f1e892a89c00217c0a513a1a1a880ff12d1eda3a3abaa553ffe7617f285c30a1/resolv.conf" does not exist
E1001 12:51:31.983385       1 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "helm-install-traefik-hbldm_kube-system(099898f5-e44a-11e9-9ee8-0242c0a87002)" failed: rpc error: code = Unknown desc = rewrite resolv.conf failed for pod "helm-install-traefik-hbldm": ResolvConfPath "/var/lib/docker/containers/f1e892a89c00217c0a513a1a1a880ff12d1eda3a3abaa553ffe7617f285c30a1/resolv.conf" does not exist
E1001 12:51:31.983416       1 kuberuntime_manager.go:697] createPodSandbox for pod "helm-install-traefik-hbldm_kube-system(099898f5-e44a-11e9-9ee8-0242c0a87002)" failed: rpc error: code = Unknown desc = rewrite resolv.conf failed for pod "helm-install-traefik-hbldm": ResolvConfPath "/var/lib/docker/containers/f1e892a89c00217c0a513a1a1a880ff12d1eda3a3abaa553ffe7617f285c30a1/resolv.conf" does not exist
E1001 12:51:31.983466       1 pod_workers.go:190] Error syncing pod 099898f5-e44a-11e9-9ee8-0242c0a87002 ("helm-install-traefik-hbldm_kube-system(099898f5-e44a-11e9-9ee8-0242c0a87002)"), skipping: failed to "CreatePodSandbox" for "helm-install-traefik-hbldm_kube-system(099898f5-e44a-11e9-9ee8-0242c0a87002)" with CreatePodSandboxError: "CreatePodSandbox for pod \"helm-install-traefik-hbldm_kube-system(099898f5-e44a-11e9-9ee8-0242c0a87002)\" failed: rpc error: code = Unknown desc = rewrite resolv.conf failed for pod \"helm-install-traefik-hbldm\": ResolvConfPath \"/var/lib/docker/containers/f1e892a89c00217c0a513a1a1a880ff12d1eda3a3abaa553ffe7617f285c30a1/resolv.conf\" does not exist"

Per a conversation in slack, the resolv.conf issue is resolvable by mounting the host /var/lib/docker/containers location, but does not resolve the /proc issues.

Scope of your request

I would like explicit support for running k3s with docker enabled through k3d, including example docs.

Describe the solution you'd like

A new CLI argument should be added to create (e.g. k3d create --docker) that automatically injects the correct server/agent args and launches with a valid default image. The user may need to provide additional information (e.g., a docker socket volume or a DOCKER_HOST) if relevant.

This seems like it would necessitate an explicit k3s-dind image. The new image would likely need to be based out of rancher/k3s.

Describe alternatives you've considered

I've found one implementation available for running k3s with dind, but does not provide as clean a workflow as k3d. I also have not been able to get this to launch correctly. It suffers similar issues as above if ran with the host docker socket mounted (and different issues otherwise).

[rpocase]

I was able to get a functional solution with some help from the k3s-dind maintainer in unbounded-systems/k3s-dind#4. I've provided some context for my requirements and rough solution below.

To give some context on why I needed this - I am working on getting Hyperledger Fabric set up within a Kubernetes environment. The way Fabric installs chaincode (smart contracts) specifically requires docker support on "peer" nodes, for the time being. My CI environment is GitLab, which heavily emphasizes running jobs within (docker) containers. I wanted a pattern that easily enables execution in that environment and locally for testing changes to the deployment. When researching local k8s development, k3s + k3d seemed like a natural fit (and was VERY easy to deploy in GitLab without the --docker requirement).

The K3D side:
I am not using the linked k3s-dind directly - I actually created a in house image that is very close, but provides a docker image API that k3d expects.

E.g.,

FROM unboundedsystems/k3s-dind:latest
# overwrite upstream start-k3s.sh with your wrapper
COPY ./start-k3s.sh /usr/local/bin/start-k3s.sh
ENTRYPOINT ["/usr/local/bin/start-k3s.sh"]
CMD ["agent"]

The replaced start-k3s.sh will look very similar to the upstream variant, but with the last command being starting the k3s executable (where k3d ultimately injects input)

...
runDocker
k3s "$@"

From there, k3d can be used to launch a cluster just by specifying the right server/agent args and your custom image:

k3d create --workers 2 \
      --server-arg '--docker' --agent-arg '--docker' \
      --image your-k3d-dind:latest

I think I am safe to launch my k3s server without docker support, but keeping the environments consistent for now.

This worked great locally, but took some tinkering to get working in CI. I ended leveraging GitLab's DIND service pattern but launching another dind for direct interaction with that docker daemon, as opposed to my Gitlab DIND service docker daemon. E.g.

gitlab-job:
   stage: my-stage
   image: docker:stable
   services: docker:stable-dind
   before_script:
       - docker run --rm -d --name k3d  docker:dind
   script:
       - docker exec k3d sh -c 'k3d create --workers 2 --server-arg '--docker' --agent-arg '--docker' --image your-k3d-dind:latest'
   after_script:
       - docker stop k3d

Using a similar pattern, I've been able to get 3 layers of containerization working for my requirements (gitlab docker executor (1) -> dind (2) -> k3s pods running docker containers (3) -> chaincode running as sibling containers within pods).

I would ultimately be ok with closing this as a niche issue. It would be nice to have an upstream k3d compatible dind image, but wrapping another dind solution is was low enough fruit for me.

[iwilltry42]

Thanks a lot for this writeup @rpocase 😃
It's really helpful and now I finally fully understand what you were trying to achieve :D Good job!

I'm not sure, where a k3s-dind image would fit, but probably, it'll belong into rancher/k3s.

I think, if we have a website for this project at some point, your text (+ some extras) would be a great fit for a "use-cases" section. What do you think? Would it be OK for you to put this there?

Additionally I'd like to ask you for a bit more info on how you now do testing on the k3d cluster that you create in the pipeline? How do you get test results out of there?

Thanks again :)

[rpocase]

@iwilltry42 Apologies for the delay - totally forgot about your questions.

I am absolutely all right with using my text as a baseline for use case documentation.

As for test results - I'm not at the point where I am exporting results from the cluster other than exit codes (and stdout in the pipeline/local worfklow). Exporting test result files from the cluster will come once kubernetes is integrated into our devs primary workflow. This has taken a backseat for various reasons.

[micw]

Hello,

I spent a few hours to get the following setup to work. It's not directly releated to k3d but I occured similar issues, so my solution might help you here.

I run k3s on my machine in docker (for easier maintainability). I want to run it with --docker mode (required when I want to run pipeworks on it which I need for my a-bit-special home setup). And I want it to use the hosts docker. That means my k3s uses that same docker that it runs on.

I achieved it using this compose file:

version: '3.3'
services:

  server:
    image: "rancher/k3s:${K3S_VERSION:?err}"
    network_mode: host
    pid: host
    command:
    - server
    - --https-listen-port=6443
    - --default-local-storage-path=/data/k8s/volumes
    - --docker
    tmpfs:
    - /run
    ulimits:
      nproc: 65535
      nofile:
        soft: 65535
        hard: 65535
    privileged: true
    restart: unless-stopped
    environment:
    - K3S_TOKEN=${K3S_TOKEN:?err}
    - K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
    - K3S_KUBECONFIG_MODE=666
    volumes:
    - ./k3s-server:/var/lib/rancher/k3s
    - /data:/data
    - /var/run/docker.sock:/var/run/docker.sock
    - type: bind
      source: /var/lib/docker
      target: /var/lib/docker
      bind:
        propagation: rshared
    - type: bind
      source: /var/lib/kubelet
      target: /var/lib/kubelet
      bind:
        propagation: rshared   
    - /sys:/sys
    # This is just so that we get the kubeconfig file out
    - .:/output

network_mode: host - required for me so that containers can do stuff with my host network (e.g. add IP addresses via pipeworks)

pid: host - solves the access to /prod issues

/var/lib/docker and /var/lib/kubelet must be mounted with rshared propagation because the kubelet does some cross-bind-mounts that needs to be available to pods running on docker, so it must be propagated to the host.

Kind regards,
Michael.

[Reintjuu]

@micw, thanks for your efforts. I'd like to get this working as well. But I'm getting an error, and I don't quite understand how to set up my volumes correctly:

Cannot create container for service server: path /var/lib/kubelet is mounted on / but it is not a shared mount.

Sometimes I get the same error for the Docker mount:

Cannot create container for service server: path /var/lib/docker is mounted on / but it is not a shared mount.

What else did you configure to make this work, if I may ask?