Building/deploying pods on 4.8.0-0.okd-2021-10-24-061736 via git fails with memory cgroup limits error

[markandrewj]

Hello,

We recently provisioned a OKD 4.7 cluster using the bare metal UPI method, and then upgraded the cluster to OKD 4.8. Currently when trying to build a pod using Git as the source via the catalog, or a custom git repository, we receive the following error.

Adding cluster TLS certificate authority to trust store
Caching blobs under "/var/cache/blobs".
error: failed to retrieve cgroup limits: cannot determine cgroup limits: open /sys/fs/cgroup/memory/memory.limit_in_bytes: no such file or directory

The nodes are virtual machines on top of RHV. Each of the worker nodes is provisioned with 12G of memory.

[root@worker0 ~]# free
               total        used        free      shared  buff/cache   available
Mem:        12242308     2730344      286172       57624     9225792     9097876
Swap:              0           0           0

I notice that /sys/fs/cgroup looks considerably different between our existing OpenShift 4.7 clusters and the new OKD 4.8 cluster. This may be a difference between Fedora CoreOS and RedHat CoreOS.

Fedora CoreOS version:

[root@worker0 ~]# cat /etc/redhat-release
Fedora release 34 (Thirty Four)

OKD 4.8 /sys/fs/cgroup:

[root@worker0 ~]# ll /sys/fs/cgroup
total 0
-r--r--r--.  1 root root 0 Nov 16 18:28 cgroup.controllers
-rw-r--r--.  1 root root 0 Nov 16 18:30 cgroup.max.depth
-rw-r--r--.  1 root root 0 Nov 16 18:30 cgroup.max.descendants
-rw-r--r--.  1 root root 0 Nov 16 18:28 cgroup.procs
-r--r--r--.  1 root root 0 Nov 16 18:30 cgroup.stat
-rw-r--r--.  1 root root 0 Dec  1 20:02 cgroup.subtree_control
-rw-r--r--.  1 root root 0 Nov 16 18:30 cgroup.threads
-rw-r--r--.  1 root root 0 Nov 16 18:30 cpu.pressure
-r--r--r--.  1 root root 0 Nov 16 18:30 cpu.stat
-r--r--r--.  1 root root 0 Nov 16 18:30 cpuset.cpus.effective
-r--r--r--.  1 root root 0 Nov 16 18:30 cpuset.mems.effective
drwxr-xr-x.  2 root root 0 Nov 16 18:30 dev-hugepages.mount
drwxr-xr-x.  2 root root 0 Nov 16 18:30 dev-mqueue.mount
drwxr-xr-x.  2 root root 0 Nov 16 18:28 init.scope
-rw-r--r--.  1 root root 0 Nov 16 18:30 io.cost.model
-rw-r--r--.  1 root root 0 Nov 16 18:30 io.cost.qos
-rw-r--r--.  1 root root 0 Nov 16 18:30 io.pressure
-rw-r--r--.  1 root root 0 Nov 16 18:30 io.prio.class
-r--r--r--.  1 root root 0 Nov 16 18:30 io.stat
drwxr-xr-x.  4 root root 0 Nov 16 18:30 kubepods.slice
drwxr-xr-x.  2 root root 0 Nov 16 18:30 machine.slice
-r--r--r--.  1 root root 0 Nov 16 18:30 memory.numa_stat
-rw-r--r--.  1 root root 0 Nov 16 18:30 memory.pressure
-r--r--r--.  1 root root 0 Nov 16 18:30 memory.stat
-r--r--r--.  1 root root 0 Nov 16 18:30 misc.capacity
drwxr-xr-x.  2 root root 0 Nov 16 18:30 sys-fs-fuse-connections.mount
drwxr-xr-x.  2 root root 0 Nov 16 18:30 sys-kernel-config.mount
drwxr-xr-x.  2 root root 0 Nov 16 18:30 sys-kernel-debug.mount
drwxr-xr-x.  2 root root 0 Nov 16 18:30 sys-kernel-tracing.mount
drwxr-xr-x. 33 root root 0 Dec  1 20:02 system.slice
drwxr-xr-x.  3 root root 0 Dec  1 20:01 user.slice

OpenShift 4.7 /sys/fs/cgroup:

[root@worker0 ~]# ll /sys/fs/cgroup
total 0
dr-xr-xr-x. 7 root root  0 Nov 18 21:35 blkio
lrwxrwxrwx. 1 root root 11 Nov 18 21:35 cpu -> cpu,cpuacct
dr-xr-xr-x. 7 root root  0 Nov 18 21:35 cpu,cpuacct
lrwxrwxrwx. 1 root root 11 Nov 18 21:35 cpuacct -> cpu,cpuacct
dr-xr-xr-x. 4 root root  0 Nov 18 21:35 cpuset
dr-xr-xr-x. 7 root root  0 Nov 18 21:35 devices
dr-xr-xr-x. 4 root root  0 Nov 18 21:35 freezer
dr-xr-xr-x. 4 root root  0 Nov 18 21:35 hugetlb
dr-xr-xr-x. 7 root root  0 Nov 18 21:35 memory
lrwxrwxrwx. 1 root root 16 Nov 18 21:35 net_cls -> net_cls,net_prio
dr-xr-xr-x. 4 root root  0 Nov 18 21:35 net_cls,net_prio
lrwxrwxrwx. 1 root root 16 Nov 18 21:35 net_prio -> net_cls,net_prio
dr-xr-xr-x. 4 root root  0 Nov 18 21:35 perf_event
dr-xr-xr-x. 7 root root  0 Nov 18 21:35 pids
dr-xr-xr-x. 2 root root  0 Nov 18 21:35 rdma
dr-xr-xr-x. 7 root root  0 Nov 18 21:35 systemd

I am not sure if this is a bug, or if there is day two configuration we have missed. We have provisioned several instances of OpenShift before provisioning this OKD cluster though and this is the first time encountering the issue. If you would like the output of systemd-cgls, or any other information, please let me know.

[vrutkovs]

cgroupsv2 has been enabled, which is supported only from OKD 4.9

Please attach (or upload to the public file sharing service) must-gather archive

[markandrewj]

Thank you for the quick response. We had snapshots of the virtual machines before upgrading to OKD 4.8, so we reverted the nodes back to 4.7 and checked to see if the issue was still present. Even under 4.7 cgroup's v2 was active. We upgraded from 4.7 to 4.8, instead of installing 4.8 directly, because we currently have two OpenShift 4.7 clusters we are planning upgrades for. We wanted to test out the process and explore what is new in 4.8.

My colleague tracked down 4.8.0-0.okd-2021-10-24-061736 and 4.7.0-0.okd-2021-09-19-013247 as the underlying images.

Is there a way to provide you the must-gather in private. Since we are using this in the enterprise space I want to make sure nothing gets leaked. Do you have any idea how cgroup's v2 may have been enabled for the OKD 4.7/4.8 images? Or if cgroup's v2 being enabled was intentional, why it is not working properly?

Just as a note, the reason we are using OKD instead of OpenShift for testing is because RedHat said they were not willing to provide free to use licenses for a test cluster. In our case the test cluster is for administrators, and not for our developers to test applications. With most of our on premises products we stand up at least three instances, test, development, and production. We then use the test environment to validate there are no issues with upgrades or major changes before taking the same steps under development and production. It was suggested that we use cloud resources to spin up a test cluster on demand, but it wouldn't align with the our on premise installation and configuration, so it would not make for a relevant test environment. Do you know if it is possible to run OpenShift unlicensed? My impression is it still works fine, but we would not get support for the unlicensed cluster, which is not required for our use case. I am asking because we have found OKD to be less stable then OpenShift.

[LorbusChris]

Is this reproducible in OKD 4.9?

Regarding OCP: Please take a look at https://www.redhat.com/en/technologies/cloud-computing/openshift/try-it
Unlicensed OCP clusters will not be able to receive updates.

[markandrewj]

Hello @LorbusChris, I am not sure if this is reproducible in 4.9. We could possibly upgrade OKD to 4.9 as a test, but we really want to keep OKD in lockstep with the version our other OpenShift clusters are on. As I mentioned the intention of the OKD cluster is to try and catch any potential breaking changes with upgrades, or to test high risk cluster configuration changes, in an environment where we don't have to worry about impacting users or developers. The other OpenShift clusters are actually on 4.7 at the moment, be will be bumping them up to 4.8 after the holidays. I am guessing that cgroups v2 is better supported in 4.9 and would work properly. What I don't understand is how we ended up with this problem under both 4.7 and 4.8 of OKD. I understand that k8s and OpenShift are transitioning to the use of cgroups v2, but this is not an issue we have had while using the OpenShift images. We also had to struggle to get the OKD cluster up and running (#897). Thank you for the the information about licensing, it's a bit unfortunate though, most of the products we use give us a couple extra licenses to run a test/backup instance.

[LorbusChris]

Have you tried to manually set cgroupsv1 by adding a MachineConfig that sets the respective karg?

[markandrewj]

We haven't tried this yet, but we can give it a go. We haven't had to do this for any other clusters so far though, so I am wondering why it is necessary this time.

[vrutkovs]

Looks similar to #710

[LorbusChris]

Yes I think that's the same issue. Adding kargs via the kernel-args file wasn't really ever resolved, but since that file interface is considered deprecated now, and we've moved to setting kargs via the MachineConfig field in 4.9, we should be good going forward. The underlying issue here was that FCOS moved to cgroupsv2 some time ago, and our config to force it back to cgroupsv1 via kernel-args wasn't picked up properly.

Users who want to go back to cgroupsv1 can add a MachineConfig object like this to do that:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-okd-worker-cgroupsv1
spec:
  config:
    ignition:
      version: 3.2.0
  kernelArguments:
    - systemd.unified_cgroup_hierarchy=0

[markandrewj]

Thank you both for the follow up information. I will try creating the MachineConfig object you have suggested and let you know the result. I will also check what the kernel argument was set to before making changes.

[markandrewj]

@LorbusChris we applied the MachineConfig object that you provided and it resolved our issue. I checked for the kernel parameter before applying the change, but I checked using sysctl -a, I didn't notice that kernel parameter was being set via a grub command line argument (/proc/cmdline) until after applying the object.

I am not sure if this was caused by the images we used, or all the 4.7/4.8 images are like this. Fixing the images would be ideal if it is relevant to all of them, but if it is not possible to fix the images I think this should be added to the documentation. Thanks for your help.

[vrutkovs]

Perhaps you started with FCOS which defaults to cgroupsv2, while OKD 4.7 was tested with cgroupsv1-by-default FCOS, so implicit setting was required

[markandrewj]

I am thinking that the FCOS image we used, even though it was an older image, defaulted to cgroups v2 as well. I know it is becoming less common to install 4.7 or 4.8, but I am just thinking maybe this MachineConfig object should be created by default for older versions of OKD, or there should be some documentation about how to resolve the issue under older versions of OKD if it occurs.

It sounds like there was a fix applied for this in the issue you referenced, but for specific images. I am not really sure what is better. From an upgrade perspective it is probably better to be setting the kernel parameter in the image, and then providing some reference documentation in case someone like myself still encounters the issue. I am happy this was relatively easy to identify and fix. We can probably mark this issue as closed, but I wanted to follow up with some feedback before doing so.

[LorbusChris]

OKD releases on a rolling basis, so there won't usually be any more releases for 4.7 or 4.8 now. Feel free to open a docs PR with clarification against https://github.com/openshift/openshift-docs.