docker cgroup driver discussion - cgroupfs or systemd

[euank]

Background

CoreOS currently ships docker with a non-default configuration of --exec-opt native.cgroupdriver=systemd. This option manages Docker's container's cgroups with systemd instead of the cgroupfs driver.

I don't know the full history for why this configuration was chosen, but I believe it was related to interacting well with Kubernetes.

Problem

I believe that this change might no longer make sense for CoreOS to ship. Below, I've enumerated reasons why we might want to use each driver:

Reasons to use cgroupfs

• Better tested with docker (and related projects like k8s, cadvisor) due to being the default upstream driver ⭐⭐
• Doesn't mess up handling of init.scope in systemd 226+ (current issue in CoreOS alpha/beta) (runc issue, cadvisor issue)
• First-supported for some Kubernetes features (e.g. pod-level cgroups)
• Jessie Frazelle's endorsement 🌟

(important reasons starred)

Reasons to use systemd

• Backwards compatibility with the cgroup layout
• Better integration with some tools (are there any worth noting? I'm specifically thinking of systemctl status but I'm not sure that actually does integrate better)
• Belief that having systemd manage cgroups is fundamentally better than having dockerd/runc do so

Conclusion

I can't claim that any of the reasons for either choice make one inarguably better, but I think there are enough issues here that it's worth having the discussion. This will also help guide some related issues (how effort is directed on some K8s + CoreOS issues etc).

cc @philips @mischief @vishh @timstclair @crawford @marineam @aaronlevy

[crawford]

The original reason for the change was due to the poor interaction between Docker and cAdvisor. If I remember correctly, the "docker" cgroup driver was creating a hierarchy that cAdvisor didn't recognize. If that is in fact the issue, then I think we can reverse that configuration change.

[peebs]

I've spent a little time testing the docker cgroup driver on docker on the latest coreos alpha with the v1.3 coreos-kubernetes repo. So far I havn't been able to reproduce some of the possibly related issues:
coreos/coreos-kubernetes#293
kubernetes/kubernetes#12533

So I don't have enough to make the call on this but next step is just running e2e tests against a cluster setup with coreos-kubernetes on coreos and with the docker cgroup driver. I'll be out of office for a few days starting now.

possibly related: kubernetes/kubernetes#27383 (comment)

[peebs]

Spent some more time with this and talked to some people. I think the consensus from our team is to ship it. No tests I've been able to run have brought issues nor have I seen anything in the logs that worry me or at least differ from the systemd driver. I've done this testing on a recent alpha and kubernetes v1.3.4.

[manojlds]

Hi, how can I know if this has made its way into CoreOS stable?

[zihaoyu]

@manojlds Last time I tried latest stable it's not there.

[crawford]

@manojlds it can be a bit of a rat's nest to follow, but I'll show you how I figured it out in this case. If you look at my pull request which fixed the issue, you'll notice that Docker was renamed from docker-1.11.2 to docker-1.11.2-r1. So docker-1.11.2-r1 is the first version with this fix. If you look at the package list for Stable, you'll see that the current Stable (as of this writing) ships docker-1.10.3-r5. So it's not there yet. The package list for Beta, however, shows docker-1.11.2-r3. So Beta would have the fix.

[stensonb]

the current beta channel build (1185.1.0) includes docker-1.11.2-r5, fyi...

[redbaron]

what is the current status of this? for docker 1.12.6 and systemd 231, which cgroup driver should be used by docker and kubelet?

[crawford]

docker's default, cgroupfs, is now used.

[redbaron]

I found that without switching both docker and kubelet to systemd, kubelet checks and complains about non-existing containers, or maybe containers it can't reach.

[mattshma]

Could somebody tell us which cgroup driver should be used by docker and kubelet please? 😄

[ChaitanyaVSH]

how to change the Cgroup in docker from Systemd to cgroupfs

[psegaro]

Setup daemon.

cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=cgroupfs"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

Restart docker.

systemctl daemon-reload
systemctl restart docker

From https://kubernetes.io/docs/setup/cri/

[mars64]

Sorry to resurrect the dead here but this thread seems to disagree with what appears to be a more recent consensus and I'd love to know more about what the recommended go-forward configurations are. As mentioned in kubernetes/kubeadm#1394 - we are indeed seeing situations where nodes that are configured to use cgroupfs for kubelet and docker and systemd for the rest can become unstable under resource pressure, currently on 2079.5.1, but will update shortly and test again. Using systemd on systems that support it appears to be recommended by Kubernetes setup docs: https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cgroup-drivers

Have the cgroup driver configuration recommendations changed since this thread? What is the current recommendation (or) is there a different place to find more information on cgroup driver configuration recommendations for CoreOS?

[mars64]

I received some answers from the CoreOS mailing thread. An important consideration here is the fact that CoreOS is now in maintenance mode, and some otherwise critical changes like Docker upgrades simply won't happen (#2624 (comment)). EoL hasn't been announced, but it sure feels like that what this is (watch the mailing list if you're interested in this).

Regarding my above questions: cgroupfs was chosen as cgroup driver for docker for backward compatibility. Changing it should be possible but doesn't seem like there is much information out there on how to do this. At this point, it seems that if you require deeper customization of your nodes then it's prudent to consider an alternative operating system.