Add linux.resources.devices and drop linux.devices

[wking]

Spun off from my suggestions in #94. Details in the committed docs,
commit messages, and #94 discussion.

[LK4D4]

I'm don't know whether it's bad or good to force people to create devices themselves in bundle. I think all devices could be created by OCI implementation.
ping @crosbymichael @mrunalp @philips

[crosbymichael]

No I don't think it makes sense to have dev nodes created outside of the runtime

-1

[mrunalp]

👎 I think that the runtime should create the device nodes.

[wking]

On Thu, Aug 06, 2015 at 11:36:04AM -0700, Alexander Morozov wrote:

I'm don't know whether it's bad or good to force people to create
devices themselves in bundle.

I expect most users would be covered by #95 here. Once you're
creating a custom device, it seems like a wash for bundle authors
between “add an entry to the config” and “use mknod or cp”. And it's
easier for implementation authors if they don't have to support a
separate device-creation step.

[wking]

On Thu, Aug 06, 2015 at 11:40:00AM -0700, Michael Crosby wrote:

No I don't think it makes sense to have dev nodes created outside of
the runtime

Can you explain why?

[crosbymichael]

because /dev includes much more than just device nodes and we have to set it up correctly. It will also break live migration when we have to migrate and we have fd's leaking in to the container from outside the rootfs.

[wking]

On Thu, Aug 06, 2015 at 11:47:25AM -0700, Michael Crosby wrote:

because /dev includes much more than just device nodes and we have
to set it up correctly.

So all the stuff under /dev that's not a device node just needs to
show up in the "mounts" array after the mount that supplies /dev.
That doesn't sound like a big deal to me. Or is there some
particularly tricky /dev initialization that can't be handled by
ordering "mounts"?

It will also break live migration when we have to migrate and we
have fd's leaking in to the container from outside the rootfs.

I'm missing this one completely. Is the important distinction rootfs
vs. other mounts? Or are device-node file descriptors particulary
hard to preserve in a way that having runtime-created device nodes
makes easier?

[wking]

On Thu, Aug 06, 2015 at 11:57:45AM -0700, W. Trevor King wrote:

Thu, Aug 06, 2015 at 11:47:25AM -0700, Michael Crosby:

It will also break live migration when we have to migrate and we
have fd's leaking in to the container from outside the rootfs.

I'm missing this one completely. Is the important distinction
rootfs vs. other mounts? Or are device-node file descriptors
particulary hard to preserve in a way that having runtime-created
device nodes makes easier?

There's a bit more detail on this in IRC today:

10:58 < crosbymichael> most alternate suggestions will work for common
use cases, but when you start throwing user namespace support and
live migration things have to be changed
…
11:00 < crosbymichael> one bug was that for live migration, a process
had an open reference to /dev/null ( on hte host ) for it's STDIO,
and we have to close that and reopen to the local /dev/null inside
the container's rootfs or live migration does not work because there
are references to things outside the rootfs
…
11:04 < wking> hmm, I'd missed the host-vs-container /dev/null
distinction. Was it bind-mounted into the container? Why
bind-mount it instead of creating a new device in the container?
11:05 < mrunalp> mknod isn't allowed when user namespaces are enabled.
They have to bind mounted in for user namespace scenario

I don't see why having a rootfs/dev/null in your container wouldn't
work in this “I need /dev/null, but my user namespace doesn't allow me
to call mknod” case. CRIU should be able to handle external mounts
anyway 1, but that seems orthogonal to whether the device is created
by bind-mounting from the host or just already existing in the bundled
filesystem.

The permissions for mknod shouldn't matter, because the bundle author
is calling mknod (or cp) at bundle-author time, which happens before
the runtime is involved at all (and it's the runtime that sets up the
namespaces and control groups).

So I still see no technical reason that would require
config-specified, runtime-initiated mknod calls.

Also on IRC, @LK4D4 pointed out two use cases where config-defined
devices may be more convenient:

a. Running multiple containers on one rootfs, where different
containers may need different devices (e.g. only one container
needs a modem device).
b. Running one config on multiple rootfs.

For, (a) I'd probably put the modem device in the rootfs and just
ignore it (potentially adding a restrictive cgroups rule to deny it)
in the containers that didn't need it.

For (b), I'd just create all the device nodes, which might be
scriptable with something like:

$ find . -name dev -type d -execdir mknod …

But still, these cases would be more conveniently handled by
config-side device creation. However, I think making these use cases
easier isn't worth the complication of overloading mknod and cgroups
handling in the same structure (see all the conditionals in #101).

I'm less sure of the balance between these cases and a complication of
a mknod-specific section in the config. #99 is similar to this PR,
but it keeps the mknod-oriented linux.devices. If @LK4D4's use cases
are worth the spec/implementation trouble, then we probably want to
keep some form of mknod support in the spec, and on that front I'm
agnostic between the “copy from host” semantics we had before #94 and
the “specify everything mknod needs” semantics that landed with #94.

[wking]

On Fri, Aug 07, 2015 at 12:31:39PM -0700, W. Trevor King wrote:

So I still see no technical reason that would require
config-specified, runtime-initiated mknod calls.
…
But still, these cases would be more conveniently handled by
config-side device creation… I'm less sure of the balance between
these cases and a complication of a mknod-specific section in the
config.

After thinking this over this afternoon, I think dropping mknod
handling from the config is the right approach (i.e. I prefer #98 to
#99 or #94). The same use cases where @LK4D4 sees the convenience of
a spec-side mknod would also apply to any filesystem-side object, and
I think we can all agree that we don't want things like:

"files": [
{
"path": "/root/.bashrc",
"uid": 0,
"gid": 0,
"fileMode": 0644,
"content": "export HISTCONTROL=ignoreboth\n",
}
]

in the spec. So I'd rather just say “things that can be represented
in bundle filesystems do not belong in the spec”, and leave it at
that. I agree that we need cgroups device support (which was the
initial motivation for #94 1), and the approach taken in this PR
gives us that without the mknod cruft.