How to handle machine-specific config in early boot

[darkmuggle]

Moving dracutdevs/dracut#792 here to be respectful to Dracut upstream.

Axiom: Some devices require machine configuration before the OStree is mounted. This configuration is typically included during the generation of the initramfs.

In general, RHCOS users get their initramfs from Red Hat. For FCOS, user could regenerate if needed. We need a solution that works with both RHCOS and FCOS.

I have explored a number of options, each option have significant problems:

• appending or regenerating the initrd is out (due to desires to sign, don't want to mutate it). If we support appended initrd's then we open the door wide for arbitrary logic.
• reading the files from /boot alternative path was found distasteful in 01bootetc: introduce module for updating dracut etc dracutdevs/dracut#792 because it mutates the configuration at boot and provides two sources of truth.
• opening the root device and fetching a while-list is worse, especially in the case of LUKS (open, read files, close LUKS, bounce a few things, re-open....)
• using the MCO to trigger initramfs generation is trivial, however that would require the user to drop the configuration locally, assign the machine to a config pool and then enable it (the UX is rough)

Then multipath, iSCSI and other complicated disk setups can't be supported by CoreOS -- that is the state of things today.

Can we decide on what path we want to take? Or do we say, tough luck? Is there some option I haven't considered?

[darkmuggle]

/cc @dustymabe @cgwalters @jlebon Discussion from Dracut 01bootetc moved here.

[cgwalters]

In general, RHCOS users get their initramfs from Red Hat. For FCOS, user could regenerate if needed.

Note today if one enables the kernelType: realtime flag in the MCO, that turns on client-side initramfs regeneration as part of the kernel override - this gets back into the whole "should the RT kernel be a separate oscontainer/ostree ref" thing. And what gets really complicated here is in rpm-ostree we go into some gymnastics to not pull in local /etc files by default in that scenario; see coreos/rpm-ostree#1998

Now, it's certainly true that there isn't an exposed knob to regenerate the initramfs before an install today, but it'd likely work to mount the target root after coreos-installer and run rpm-ostree --sysroot=/path/to/mount initramfs --enable (though, the --sysroot path of rpm-ostree isn't heavily exercised today).

reading the files from /boot alternative path was found distasteful in dracutdevs/dracut#792 because it provides it mutates the configuration at boot and provides two sources of truth.

This is conflating the "bootetc copy" (your PR) with my proposed "just read /boot/etc/multipath.conf" though?

[darkmuggle]

This is conflating the "bootetc copy" (your PR) with my proposed "just read /boot/etc/multipath.conf" though?

No, this was in response to @jlebon concern of two sources of truth. If there are two /etc/multipath.conf (one in the root and one in boot) but different, then the initial multipath map could be different than the final one. And both could define the path for what root is mapped to. Your counter idea to bootetc by sourcing the configuration from /boot on a per-unit basis does not mitigate what Jonathan raised of two sources of truth. In fact, his concern is the reason why I moved the discussion here.

[darkmuggle]

FWIW, I started working on @cgwalters suggestion of a narrowly scoped source of specific configuration files. But I don't want to submit it until after we have made a decision here.

[cgwalters]

To reply to

I guess one other concern I have with having configs in /boot in general is that now we'd have two sources of truth: /etc (where dracut sources configs from today) and /boot/initrd-etc.

No, with the "targeted config" approach, there would only be /boot/etc/multipath.conf. With a server side generated initramfs, the dracut there wouldn't see any /etc/multipath.conf. And there'd be no reason to write a copy into the real root.

Now, if client-side initramfs regeneration is enabled, since this PR landed we still shouldn't see the copy in the real root unless the user explicitly did rpm-ostree initramfs --enable -I /etc/multipath.conf.

Right?

[jlebon]

I think this is fixed by coreos/rpm-ostree#2170 now, so tentatively closing. That said, we'll likely want more sugaring on top on the OS integration side to make it transparent and easier to use.