Console path resolution is done in host mount namespace

[cyphar]

[I realised this while trying to get the test suite to run for #774.]

The main issue is that we set up the console in the parent's mount namespace. This breaks quite a few things. In addition, --console resolution done in the parent causes things like su to not work in containers (because glibc is broken). If you have a config.json like this:

{
    "ociVersion": "0.6.0-dev",
    "platform": {
        "os": "linux",
        "arch": "amd64"
    },
    "process": {
        "terminal": true,
        "user": {},
        "args": [
            "sh"
        ],
        "env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "TERM=xterm"
        ],
        "cwd": "/",
        "capabilities": [
            "CAP_AUDIT_WRITE",
            "CAP_KILL",
            "CAP_NET_BIND_SERVICE"
        ],
        "rlimits": [
            {
                "type": "RLIMIT_NOFILE",
                "hard": 1024,
                "soft": 1024
            }
        ],
        "noNewPrivileges": true
    },
    "root": {
        "path": "rootfs",
        "readonly": true
    },
    "hostname": "runc",
    "mounts": [
        {
            "destination": "/proc",
            "type": "proc",
            "source": "proc"
        },
        {
            "destination": "/dev",
            "type": "tmpfs",
            "source": "tmpfs",
            "options": [
                "nosuid",
                "strictatime",
                "mode=755",
                "size=65536k"
            ]
        },
        {
            "destination": "/dev/pts",
            "type": "devpts",
            "source": "devpts",
            "options": [
                "nosuid",
                "noexec",
                "newinstance",
                "ptmxmode=0666",
                "mode=0620"
            ]
        },
        {
            "destination": "/dev/shm",
            "type": "tmpfs",
            "source": "shm",
            "options": [
                "nosuid",
                "noexec",
                "nodev",
                "mode=1777",
                "size=65536k"
            ]
        },
        {
            "destination": "/dev/mqueue",
            "type": "mqueue",
            "source": "mqueue",
            "options": [
                "nosuid",
                "noexec",
                "nodev"
            ]
        },
        {
            "destination": "/sys",
            "type": "none",
            "source": "/sys",
            "options": [
                "rbind",
                "nosuid",
                "noexec",
                "nodev",
                "ro"
            ]
        }
    ],
    "hooks": {},
    "linux": {
        "uidMappings": [
            {
                "hostID": 1000,
                "containerID": 0,
                "size": 1
            }
        ],
        "gidMappings": [
            {
                "hostID": 100,
                "containerID": 0,
                "size": 1
            }
        ],
        "namespaces": [
            {
                "type": "pid"
            },
            {
                "type": "ipc"
            },
            {
                "type": "uts"
            },
            {
                "type": "mount"
            },
            {
                "type": "user"
            }
        ],
        "maskedPaths": [
            "/proc/kcore",
            "/proc/latency_stats",
            "/proc/timer_stats",
            "/proc/sched_debug"
        ],
        "readonlyPaths": [
            "/proc/asound",
            "/proc/bus",
            "/proc/fs",
            "/proc/irq",
            "/proc/sys",
            "/proc/sysrq-trigger"
        ]
    }
}

/cc @crosbymichael

[cyphar]

This is probably a good reason for us to start adding integration tests for user namespaces.

[cyphar]

This happens inside (*linuxConsole).dupStdio(), because /dev/pts/ptmx is being opened there. And since it's being opened in the host, we get screwed by the mode of /dev/pts/ptmx in the host's devpts instance -- we should fix this. According to the docs, we should probably be using /dev/pts/ptmx inside the container. However, this means that we'll have to mess around with things.

[crosbymichael]

The tricky part is getting the pty master back outside of the container after opening it.

[cyphar]

This also links to several old issues with libcontainer with regard to sudo not working in containers -- it all links back to the fact that we are creating the terminal in the host. I'm trying to figure out a way to make this unnecessary.

[cyphar]

Maybe we could use unix sockets to send the fd? I'll take another look at this next week.

[crosbymichael]

i don't like unix socket sending fds. i really want to avoid these super containers with live connections going in and out.

[cyphar]

I have two ideas currently:

1. We use a unix socket, which requires setting up a unix socket just during setup. This is going to be messy because ancillary data requires a lot of C finesse. But it will work.
2. We could try forking a process that shares the set of files (~CLONE_FILES) which then uses setns(2) to join the mount (and user if available) namespace of the container to then open either /proc/1/fd/0 or /dev/pts/<number>. The issue with this is that it might not work properly (I'm trying to remember what the minimum set of clone flags are for namespaces -- I'm fairly sure you need CLONE_FLAGS). We also need to add another nsenter-like reexec thing.

[wking]

On Wed, Jun 01, 2016 at 07:37:47PM -0700, Aleksa Sarai wrote:

1. We use a unix socket, which requires setting up a unix socket
   just during setup. This is going to be messy because ancillary data
   requires a lot of C finesse. But it will work.

It's not that difficult; ~50 lines for sendfd and recvfd wrappers in
ccon, based on an example in cmsg(3).

[cyphar]

I said messy and finesse. It's not that it's hard, it's that it'll be a bit of C code we can't touch.

[wking]

On Wed, Jun 01, 2016 at 07:47:35PM -0700, Aleksa Sarai wrote:

It's not that it's hard, it's that it'll be a bit of C code we can't
touch.

Why “can't touch”? I'm not even sure what that means :p.

[wking]

On Wed, Jun 01, 2016 at 04:41:47PM -0700, Michael Crosby wrote:

i really want to avoid these super containers with live connections
going in and out.

Ccon uses an anonymous Unix socket (socketpair(2)) 1 the way runC
apparently uses a pipe 2. The host ↔ container socket gets closed
before the container process calls execve(2) (or similar) to launch
the user code. So there shouldn't be any more “live connections” than
you already have.

[cyphar]

I'm going to be honest, I didn't know about socketpair(2). This would make the code much simpler. As for "can't touch" it just means that we'll have to mix some Go and C code that has odd semantics and we have to copy from the man page for cmsg and touching it would probably cause something to blow up.

[wking]

On Wed, Jun 01, 2016 at 09:11:02PM -0700, Aleksa Sarai wrote:

… touching it would probably cause something to blow up.

That's true of most things, and what test suites are for ;).

[cyphar]

If @crosbymichael is okay with using socketpair(2) in place of pipe(2) for the parent <-> child communication, I'd be fine with implementing the code this way. But there's a question of whether or not we'll get similar issues as with the whole ECONNRESET fiasco.

[cyphar]

Wait hang on. We already use socketpair -- look at newPipe() at libcontainer/container_linux.go! So this should be a fairly simple extension.

[cyphar]

Currently working on the design plan. https://gist.github.com/cyphar/8c6b9db84fc1f2cc2d037ef07942ca83

Here's @crosbymichael's mockup of how you could implement things in a simple C program. https://gist.github.com/crosbymichael/d3045070f0e2615814aaa31e8991d7fd