Container execution stalls upon write() to binfmt procfs node

[jbeal-work]

On a new build from our ci that builds from head.

ubuntu@runner-9mkyrd3y-1622559601-394ab593:~$ sysbox-runc --version
sysbox-runc
        edition:        Community Edition (CE)
        version:        0.3.0
        commit:         df952e5276cb6e705e0be331e9a9fe88f372eab8
        built at:       Thu Apr  1 15:35:55 UTC 2021
        built by: 
        oci-specs:      1.0.2-dev
ubuntu@runner-9mkyrd3y-1622559601-394ab593:~$ docker --version
Docker version 20.10.5, build 55c4c88
ubuntu@runner-9mkyrd3y-1622559601-394ab593:~$ uname -a
Linux runner-9mkyrd3y-1622559601-394ab593 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Trying this stalls

ubuntu@runner-9mkyrd3y-1622559601-394ab593:~$ docker run -it ubuntu:18.04 bash -c  "apt-get update ; DEBIAN_FRONTEND=noninteractive apt-get install -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" -yq  binfmt-support ;  DEBIAN_FRONTEND=noninteractive apt-get install -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" -yq python"       
Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]           
Get:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]                          
Get:4 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]                                  
Get:5 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]    
Get:6 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]                                 
Get:7 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]                                     
Get:8 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]                                    
Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [31.6 kB]                   
Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [452 kB]                               
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2585 kB]                                    
Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2184 kB]    
Get:13 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [11.4 kB]  
Get:14 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [11.3 kB]        
Get:15 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2153 kB]        
Get:16 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [423 kB]
Get:17 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1413 kB]
Get:18 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [24.7 kB]
Fetched 22.7 MB in 2s (12.7 MB/s)                           
Reading package lists... Done
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libpipeline1
The following NEW packages will be installed:
  binfmt-support libpipeline1
0 upgraded, 2 newly installed, 0 to remove and 5 not upgraded.
Need to get 76.9 kB of archives.
After this operation, 281 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 libpipeline1 amd64 1.5.0-1 [25.3 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 binfmt-support amd64 2.1.8-2 [51.6 kB]
Fetched 76.9 kB in 0s (4124 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libpipeline1:amd64.
(Reading database ... 4045 files and directories currently installed.)
Preparing to unpack .../libpipeline1_1.5.0-1_amd64.deb ...
Unpacking libpipeline1:amd64 (1.5.0-1) ...
Selecting previously unselected package binfmt-support.
Preparing to unpack .../binfmt-support_2.1.8-2_amd64.deb ...
Unpacking binfmt-support (2.1.8-2) ...
Setting up libpipeline1:amd64 (1.5.0-1) ...
Setting up binfmt-support (2.1.8-2) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  file libexpat1 libmagic-mgc libmagic1 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libreadline7 libsqlite3-0 libssl1.1 mime-support python-minimal python2.7 python2.7-minimal readline-common xz-utils
Suggested packages:
  python-doc python-tk python2.7-doc binutils readline-doc
The following NEW packages will be installed:
  file libexpat1 libmagic-mgc libmagic1 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libreadline7 libsqlite3-0 libssl1.1 mime-support python python-minimal python2.7 python2.7-minimal readline-common xz-utils
0 upgraded, 17 newly installed, 0 to remove and 5 not upgraded.
Need to get 6412 kB of archives.
After this operation, 28.9 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython2.7-minimal amd64 2.7.17-1~18.04ubuntu1.6 [335 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python2.7-minimal amd64 2.7.17-1~18.04ubuntu1.6 [1291 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-minimal amd64 2.7.15~rc1-1 [28.1 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libssl1.1 amd64 1.1.1-1ubuntu2.1~18.04.9 [1301 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic/main amd64 mime-support all 3.60ubuntu1 [30.1 kB]
Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libexpat1 amd64 2.2.5-3ubuntu0.2 [80.5 kB]
Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 readline-common all 7.0-3 [52.9 kB]
Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libreadline7 amd64 7.0-3 [124 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libsqlite3-0 amd64 3.22.0-1ubuntu0.4 [499 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython2.7-stdlib amd64 2.7.17-1~18.04ubuntu1.6 [1917 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python2.7 amd64 2.7.17-1~18.04ubuntu1.6 [248 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 libpython-stdlib amd64 2.7.15~rc1-1 [7620 B]
Get:13 http://archive.ubuntu.com/ubuntu bionic/main amd64 python amd64 2.7.15~rc1-1 [140 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libmagic-mgc amd64 1:5.32-2ubuntu0.4 [184 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libmagic1 amd64 1:5.32-2ubuntu0.4 [68.6 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 file amd64 1:5.32-2ubuntu0.4 [22.1 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 xz-utils amd64 5.2.2-1.3 [83.8 kB]
Fetched 6412 kB in 0s (57.0 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libpython2.7-minimal:amd64.
(Reading database ... 4064 files and directories currently installed.)
Preparing to unpack .../00-libpython2.7-minimal_2.7.17-1~18.04ubuntu1.6_amd64.deb ...
Unpacking libpython2.7-minimal:amd64 (2.7.17-1~18.04ubuntu1.6) ...
Selecting previously unselected package python2.7-minimal.
Preparing to unpack .../01-python2.7-minimal_2.7.17-1~18.04ubuntu1.6_amd64.deb ...
Unpacking python2.7-minimal (2.7.17-1~18.04ubuntu1.6) ...
Selecting previously unselected package python-minimal.
Preparing to unpack .../02-python-minimal_2.7.15~rc1-1_amd64.deb ...
Unpacking python-minimal (2.7.15~rc1-1) ...
Selecting previously unselected package libssl1.1:amd64.
Preparing to unpack .../03-libssl1.1_1.1.1-1ubuntu2.1~18.04.9_amd64.deb ...
Unpacking libssl1.1:amd64 (1.1.1-1ubuntu2.1~18.04.9) ...
Selecting previously unselected package mime-support.
Preparing to unpack .../04-mime-support_3.60ubuntu1_all.deb ...
Unpacking mime-support (3.60ubuntu1) ...
Selecting previously unselected package libexpat1:amd64.
Preparing to unpack .../05-libexpat1_2.2.5-3ubuntu0.2_amd64.deb ...
Unpacking libexpat1:amd64 (2.2.5-3ubuntu0.2) ...
Selecting previously unselected package readline-common.
Preparing to unpack .../06-readline-common_7.0-3_all.deb ...
Unpacking readline-common (7.0-3) ...
Selecting previously unselected package libreadline7:amd64.
Preparing to unpack .../07-libreadline7_7.0-3_amd64.deb ...
Unpacking libreadline7:amd64 (7.0-3) ...
Selecting previously unselected package libsqlite3-0:amd64.
Preparing to unpack .../08-libsqlite3-0_3.22.0-1ubuntu0.4_amd64.deb ...
Unpacking libsqlite3-0:amd64 (3.22.0-1ubuntu0.4) ...
Selecting previously unselected package libpython2.7-stdlib:amd64.
Preparing to unpack .../09-libpython2.7-stdlib_2.7.17-1~18.04ubuntu1.6_amd64.deb ...
Unpacking libpython2.7-stdlib:amd64 (2.7.17-1~18.04ubuntu1.6) ...
Selecting previously unselected package python2.7.
Preparing to unpack .../10-python2.7_2.7.17-1~18.04ubuntu1.6_amd64.deb ...
Unpacking python2.7 (2.7.17-1~18.04ubuntu1.6) ...
Selecting previously unselected package libpython-stdlib:amd64.
Preparing to unpack .../11-libpython-stdlib_2.7.15~rc1-1_amd64.deb ...
Unpacking libpython-stdlib:amd64 (2.7.15~rc1-1) ...
Setting up libpython2.7-minimal:amd64 (2.7.17-1~18.04ubuntu1.6) ...
Setting up python2.7-minimal (2.7.17-1~18.04ubuntu1.6) ...

Doing this on my local machine with out sysbox works fine.

[rodnymolina]

Got a fix for this one which will be merged as part of multiple changes to be pushed within the next couple of days.

The fix will prevent the behavior described above (stalling), but users within a sys-container will continue to be unable to write to binfmt_misc resources as that could potentially compromise the host. See more details here.

[jbeal-work]

Thank you

[rodnymolina]

At the end it took much longer than expected to have this one merged. Fixed as part of sysbox-fs PR #40.

[rodnymolina]

Closing now. Please reopen if have any issue.