Running KinD inside a Docker 20.10.6-dind container fails with v0.4.1

[ctalledo]

Repro steps:

1. Launch dind container:

docker run --runtime=sysbox-runc \                                                                                                                                                                                                                                                                                            
     --name dind-syscont -d \                                                                                                                                                                                                                                                                                                 
     --network some-network --network-alias docker \                                                                                                                                                                                                                                                                          
     -e DOCKER_TLS_CERTDIR=/certs \                                                                                                                                                                                                                                                                                           
     -v dind-syscont-certs-ca:/certs/ca \                                                                                                                                                                                                                                                                                     
     -v dind-syscont-certs-client:/certs/client \                                                                                                                                                                                                                                                                             
     docker:20.10.6-dind                                                                                                                                                                                                                                                                                                      

2. Launch docker cli container:

root@sysbox-test:~/nestybox/sysbox# docker run -it --rm \                                                                                                                                                                                                                                                                     
     --network some-network \                                                                                                                                                                                                                                                                                                 
     -e DOCKER_TLS_CERTDIR=/certs \                                                                                                                                                                                                                                                                                           
     -v dind-syscont-certs-client:/certs/client:ro \                                                                                                                                                                                                                                                                          
     docker:latest sh                                                                                                                                                                                                                                                                                                         

Inside the docker cli container:

3. Install KinD:

apk add curl                                                                                                                                                                                                                                                                                                                  
curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.9.0/kind-linux-amd64                                                                                                                                                                                                                                                           
chmod +x kind                                                                                                                                                                                                                                                                                                                 
cp kind /usr/bin/kind                                                                                                                                                                                                                                                                                                         

4. Create a kind cluster with kindest node:

~ #  kind create cluster --image=nestybox/kindestnode:v1.18.2                                                                                                                                                                                                                                                                 
Creating cluster "kind" ...                                                                                                                                                                                                                                                                                                   
 ✓ Ensuring node image (nestybox/kindestnode:v1.18.2) 🖼                                                                                                                                                                                                                                                                       
 ✗ Preparing nodes 📦                                                                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                                                              
ERROR: failed to create cluster: docker run error: command "docker run --hostname kind-control-plane --name kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/$
Command Output: docker: Error response from daemon: failed to copy xattrs: failed to list xattrs on /var/lib/docker/overlay2/9e0c206a7cf7a33d50da94fe146f7ed1a13604a7f3f1d448b7124d2540a461d2/merged/var/lock: no such file or directory.                                                                                     
See 'docker run --help'.                                                                                                                                                                                                                                                                                                      

[rodnymolina]

I'm not sure if this is related, but I've noticed that Sysbox is struggling to run Ubuntu 21.10 sys containers, which is also the Ubuntu release utilized by the latest KinD version (v0.11.1), so KinD might be just a red-herring here.

Try this and you should see the problem that I'm referring to right away ...

$ docker run --runtime=sysbox-runc -it --rm --name test-1 --hostname test-1 ghcr.io/nestybox/ubuntu-impish-systemd-docker:latest
...
[  OK  ] Started OpenBSD Secure Shell server.
[FAILED] Failed to start containerd container runtime.
See 'systemctl status containerd.service' for details.
[DEPEND] Dependency failed for Docker Application Container Engine.
[  OK  ] Started User Login Management.
...

If you try to run docker within this sys-container you'll see that it will crash ...

[ctalledo]

I'm not sure if this is related, but I've noticed that Sysbox is struggling to run Ubuntu 21.10 sys containers

This is not related to this issue, created issue #416 to track it.

[ctalledo]

The problem was identified as a bug in the way Sysbox intercepts and emulates the llistsetxattr() syscall.

This issue is now fixed in Sysbox top-of-tree via this commit.

The fix will be present in the Sysbox release after v0.4.1.

Closing.