lib/deploy: Add support for overlay initrds #2155

jlebon · 2020-07-24T21:43:47Z

In FCOS and RHCOS, the need to configure software in the initramfs has
come up multiple times. Sometimes, using kernel arguments suffices.
Other times, it really must be a configuration file. Rebuilding the
initramfs on the client-side however is a costly operation. Not only
does it add complexity to the update workflow, it also erodes a lot of
the value obtained from using the baked "blessed" initramfs from the
tree itself.

One elegant way to address this is to allow specifying multiple
initramfses. This is supported by most bootloaders (notably GRUB) and
results in each initrd being overlayed on top of each other.

This patch allows libostree clients to leverage this so that they can
avoid regenerating the initramfs entirely. libostree itself is agnostic
as to what kind and how much data overlay initrds contain. It's up to
the clients to enforce such boundaries.

To implement this, we add a new option to the deploy APIs called
overlay_initrds: this is an array of filepaths to CPIO archives. We
copy these files into /boot and add them to the BLS as another
initrd entry.

This command allows users to cheaply inject configuration files in the initramfs stage without having to regenerate the whole initramfs. This will be useful for configuring services involved in bringing up the root block device. ``` $ echo 'hello world' > /etc/foobar $ rpm-ostree ex initramfs-etc --track /etc/foobar Checking out tree 9a817d7... done Writing OSTree commit... done Staging deployment... done Run "systemctl reboot" to start a reboot $ rpm-ostree status State: idle Deployments: ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) BaseCommit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) InitramfsEtc: /etc/foobar Unlocked: development ● ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) $ reboot (boot into rd.break) sh-5.0# cat /etc/foobar hello world ``` See the libostree side of this at: ostreedev/ostree#2155 Lots more discussions in: coreos/fedora-coreos-tracker#94

cgwalters · 2020-07-27T20:20:13Z

There's no use case I can think of for a "pristine" ostree shipping multiple initramfses. So what if instead

We just taught the bootconfig parser to correctly preserve multiple initramfs fields (done in this code)
Downstream, projects like rpm-ostree could create e.g. /boot/rpm-ostree/custom-initramfs-foo.img and generate a new deployment using it.

IOW we think of this more like kernel arguments (which don't affect the commit checksum).

This command allows users to cheaply inject configuration files in the initramfs stage without having to regenerate the whole initramfs. This will be useful for configuring services involved in bringing up the root block device. ``` $ echo 'hello world' > /etc/foobar $ rpm-ostree ex initramfs-etc --track /etc/foobar Checking out tree 9a817d7... done Writing OSTree commit... done Staging deployment... done Run "systemctl reboot" to start a reboot $ rpm-ostree status State: idle Deployments: ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) BaseCommit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) InitramfsEtc: /etc/foobar Unlocked: development ● ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) $ reboot (boot into rd.break) sh-5.0# cat /etc/foobar hello world ``` See the libostree side of this at: ostreedev/ostree#2155 Lots more discussions in: coreos/fedora-coreos-tracker#94

This command allows users to cheaply inject configuration files in the initramfs stage without having to regenerate the whole initramfs (or even a new OSTree commit). This will be useful for configuring services involved in bringing up the root block device. ``` $ echo 'hello world' > /etc/foobar $ rpm-ostree ex initramfs-etc --track /etc/foobar Staging deployment... done Run "systemctl reboot" to start a reboot $ rpm-ostree status State: idle Deployments: ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) InitramfsEtc: /etc/foobar Unlocked: development ● ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) $ reboot (boot into rd.break) sh-5.0# cat /etc/foobar hello world ``` See the libostree side of this at: ostreedev/ostree#2155 Lots more discussions in: coreos/fedora-coreos-tracker#94

jlebon · 2020-08-13T21:44:04Z

OK, updated this now based on feedback and coreos/rpm-ostree#2170 (comment). We no longer look for the additional initramfses in the tree; it's instead passed via the new deploy APIs, checksummed, copied into /boot, and added into the BLS entry. The end result is that this is now an even more lightweight operation for clients!

This command allows users to cheaply inject configuration files in the initramfs stage without having to regenerate the whole initramfs (or even a new OSTree commit). This will be useful for configuring services involved in bringing up the root block device. ``` $ echo 'hello world' > /etc/foobar $ rpm-ostree ex initramfs-etc --track /etc/foobar Staging deployment... done Run "systemctl reboot" to start a reboot $ rpm-ostree status State: idle Deployments: ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) InitramfsEtc: /etc/foobar Unlocked: development ● ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) $ reboot (boot into rd.break) sh-5.0# cat /etc/foobar hello world ``` See the libostree side of this at: ostreedev/ostree#2155 Lots more discussions in: coreos/fedora-coreos-tracker#94

jlebon · 2020-08-17T13:51:43Z

Split out the trivial/prep patches into #2172.

Miscellaneous patches split out of #2155

src/libostree/ostree-sysroot-deploy.c

jlebon · 2020-09-28T19:06:44Z

Added an installed tests and marked ready for review!

Requires: #2204

Edit: I also updated coreos/rpm-ostree#2170 to show how the new APIs are used.

This command allows users to cheaply inject configuration files in the initramfs stage without having to regenerate the whole initramfs (or even a new OSTree commit). This will be useful for configuring services involved in bringing up the root block device. ``` $ echo 'hello world' > /etc/foobar $ rpm-ostree ex initramfs-etc --track /etc/foobar Staging deployment... done Run "systemctl reboot" to start a reboot $ rpm-ostree status State: idle Deployments: ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) InitramfsEtc: /etc/foobar ● ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) $ reboot (boot into rd.break) sh-5.0# cat /etc/foobar hello world ``` See the libostree side of this at: ostreedev/ostree#2155 Lots more discussions in: coreos/fedora-coreos-tracker#94

Prep for actually teaching the rest of the codebase about this. We keep the primary initrd in the `options` hash table for backwards compatibility.

And make the `override_kernel_argv` one of those options. This is mostly a mechanical move here, no functional change otherwise. Prep for adding a new option.

In FCOS and RHCOS, the need to configure software in the initramfs has come up multiple times. Sometimes, using kernel arguments suffices. Other times, it really must be a configuration file. Rebuilding the initramfs on the client-side however is a costly operation. Not only does it add complexity to the update workflow, it also erodes a lot of the value obtained from using the baked "blessed" initramfs from the tree itself. One elegant way to address this is to allow specifying multiple initramfses. This is supported by most bootloaders (notably GRUB) and results in each initrd being overlayed on top of each other. This patch allows libostree clients to leverage this so that they can avoid regenerating the initramfs entirely. libostree itself is agnostic as to what kind and how much data overlay initrds contain. It's up to the clients to enforce such boundaries. To implement this, we add a new ostree_sysroot_stage_overlay_initrd which takes a file descriptor and returns a checksum. Then users can pass these checksums when calling the deploy APIs via the new array option `overlay_initrds`. We copy these files into `/boot` and add them to the BLS as another `initrd` entry.

jlebon · 2020-09-30T17:30:07Z

Rebased this now that #2204 is merged!

cgwalters

One comment, otherwise I think this is looking good!

cgwalters · 2020-10-01T17:16:49Z

src/libostree/ostree-sysroot.c

+  for (char **it = overlay_initrds; it && *it; it++)
+    {
+      const char *basename = glnx_basename (*it);
+      if (strlen (basename) != (_OSTREE_SHA256_STRING_LEN + strlen (".img")))


It'd be nice to allow people to have a "short name" for their overlay initrd to describe its purpose without needing to unpack it to look.

Basically we append the checksum to the name they provide?

IOW it's /boot/ostree/initramfs-overlays/kdump-41af286dc0b172ed2f1ca934fd2278de4a1192302ffa07087cea2682e7d372e3.img and not just /boot/ostree/initramfs-overlays/41af286dc0b172ed2f1ca934fd2278de4a1192302ffa07087cea2682e7d372e3.img

If you prefer we can do this as a followup too.

Are you suggesting expanding the overlay initrd staging API as e.g.:

diff --git a/src/libostree/ostree-sysroot.h b/src/libostree/ostree-sysroot.h index 3a3b6a77..ef5a353c 100644 --- a/src/libostree/ostree-sysroot.h +++ b/src/libostree/ostree-sysroot.h @@ -189,6 +189,7 @@ gboolean ostree_sysroot_write_deployments_with_options (OstreeSysroot *self, _OSTREE_PUBLIC gboolean ostree_sysroot_stage_overlay_initrd (OstreeSysroot *self, int fd, + const char short_id, char **out_checksum, GCancellable *cancellable, GError **error);

?

Hmm, or maybe we could change the semantics of OstreeSysrootDeployTreeOpts.overlay_initrds to have the checksum be optionally prefixed by an ID? (E.g. my-id:<sha256>).

🤔 Honestly, while the code here does support multiple overlays, I think in practice it's not something that will be used often. E.g. in the rpm-ostree use case, there is just a single "initramfs-etc" initrd, and my hope is that it's the one way through which users interact with this feature.

I think I'd vote for leaving the API simple and we can do the second approach mentioned above if it comes up? (Because the first approach would obviously be an API break.)

OK, fine by me!

cgwalters

/lgtm

openshift-ci-robot · 2020-10-02T20:28:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, jlebon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [cgwalters,jlebon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This is a basic `.packit.yaml` integration file which will allow us have continuous builds of OSTree in cosa and upstream CI. If things go well, we'll likely deploy this in other build tools like rpm-ostree. Prompted by wanting to get ostreedev#2155 out to unblock coreos/rpm-ostree#2170.

This command allows users to cheaply inject configuration files in the initramfs stage without having to regenerate the whole initramfs (or even a new OSTree commit). This will be useful for configuring services involved in bringing up the root block device. ``` $ echo 'hello world' > /etc/foobar $ rpm-ostree ex initramfs-etc --track /etc/foobar Staging deployment... done Run "systemctl reboot" to start a reboot $ rpm-ostree status State: idle Deployments: ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) InitramfsEtc: /etc/foobar ● ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) $ reboot (boot into rd.break) sh-5.0# cat /etc/foobar hello world ``` See the libostree side of this at: ostreedev/ostree#2155 Lots more discussions in: coreos/fedora-coreos-tracker#94

This command allows users to cheaply inject configuration files in the initramfs stage without having to regenerate the whole initramfs (or even a new OSTree commit). This will be useful for configuring services involved in bringing up the root block device. ``` $ echo 'hello world' > /etc/foobar $ rpm-ostree ex initramfs-etc --track /etc/foobar Staging deployment... done Run "systemctl reboot" to start a reboot $ rpm-ostree status State: idle Deployments: ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) InitramfsEtc: /etc/foobar ● ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) $ reboot (boot into rd.break) sh-5.0# cat /etc/foobar hello world ``` See the libostree side of this at: ostreedev/ostree#2155 Lots more discussions in: coreos/fedora-coreos-tracker#94 Closes: coreos#1930

This command allows users to cheaply inject configuration files in the initramfs stage without having to regenerate the whole initramfs (or even a new OSTree commit). This will be useful for configuring services involved in bringing up the root block device. ``` $ echo 'hello world' > /etc/foobar $ rpm-ostree ex initramfs-etc --track /etc/foobar Staging deployment... done Run "systemctl reboot" to start a reboot $ rpm-ostree status State: idle Deployments: ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) InitramfsEtc: /etc/foobar ● ostree://fedora:fedora/x86_64/coreos/testing-devel Version: 32.20200716.dev.1 (2020-07-16T02:47:29Z) Commit: 9a817d75bef81b955179be6e602d1e6ae350645b6323231a62ba2ee6e5b9644b GPGSignature: (unsigned) $ reboot (boot into rd.break) sh-5.0# cat /etc/foobar hello world ``` See the libostree side of this at: ostreedev/ostree#2155 Lots more discussions in: coreos/fedora-coreos-tracker#94 Closes: #1930

openshift-ci-robot requested review from d4s and mwleeds July 24, 2020 21:43

openshift-ci-robot added the approved label Jul 24, 2020

jlebon mentioned this pull request Jul 24, 2020

Add new ex initramfs-etc command coreos/rpm-ostree#2170

Merged

jlebon marked this pull request as draft July 25, 2020 14:31

openshift-ci-robot added the do-not-merge/work-in-progress label Jul 25, 2020

jlebon force-pushed the pr/add-initrds branch 2 times, most recently from 5dc1f01 to 58b1f03 Compare August 13, 2020 21:32

jlebon force-pushed the pr/add-initrds branch from 58b1f03 to 0cb02d0 Compare August 13, 2020 21:41

jlebon force-pushed the pr/add-initrds branch from 0cb02d0 to 399c880 Compare August 14, 2020 16:14

jlebon force-pushed the pr/add-initrds branch from 399c880 to 4aacfb4 Compare August 14, 2020 18:32

jlebon changed the title ~~lib/deploy: Add support for additional initrds~~ lib/deploy: Add support for overlay initrds Aug 14, 2020

jlebon force-pushed the pr/add-initrds branch from 4aacfb4 to d0bc766 Compare August 17, 2020 13:49

openshift-merge-robot added a commit that referenced this pull request Aug 17, 2020

Merge pull request #2172 from jlebon/pr/add-initrds-prep

364556b

Miscellaneous patches split out of #2155

jlebon force-pushed the pr/add-initrds branch from d0bc766 to 5cce5f2 Compare August 17, 2020 19:28

cgwalters reviewed Aug 18, 2020

View reviewed changes

src/libostree/ostree-sysroot-deploy.c Outdated Show resolved Hide resolved

cgwalters mentioned this pull request Sep 3, 2020

create_disk: Use bootupd to install uefi if configured coreos/coreos-assembler#1695

Merged

cgwalters mentioned this pull request Sep 15, 2020

How to enable kdump feature on Fedora CoreOS coreos/fedora-coreos-tracker#622

Closed

jlebon force-pushed the pr/add-initrds branch from 5cce5f2 to 83c8505 Compare September 28, 2020 19:05

jlebon marked this pull request as ready for review September 28, 2020 19:05

openshift-ci-robot removed the do-not-merge/work-in-progress label Sep 28, 2020

jlebon added 3 commits September 30, 2020 13:29

lib/bootconfig: Add support for multiple initrd keys

f7500bb

Prep for actually teaching the rest of the codebase about this. We keep the primary initrd in the `options` hash table for backwards compatibility.

lib/deploy: Add deploy/stage APIs with options

40fea4c

And make the `override_kernel_argv` one of those options. This is mostly a mechanical move here, no functional change otherwise. Prep for adding a new option.

jlebon force-pushed the pr/add-initrds branch from 83c8505 to 81b13da Compare September 30, 2020 17:29

cgwalters approved these changes Oct 1, 2020

View reviewed changes

cgwalters approved these changes Oct 2, 2020

View reviewed changes

openshift-ci-robot assigned cgwalters Oct 2, 2020

openshift-ci-robot added the lgtm label Oct 2, 2020

openshift-merge-robot merged commit 06a77bf into ostreedev:master Oct 2, 2020

jlebon mentioned this pull request Oct 6, 2020

Add Packit integration #2210

Merged

jlebon deleted the pr/add-initrds branch April 24, 2023 02:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lib/deploy: Add support for overlay initrds #2155

lib/deploy: Add support for overlay initrds #2155

jlebon commented Jul 24, 2020 •

edited

Loading

cgwalters commented Jul 27, 2020

jlebon commented Aug 13, 2020 •

edited

Loading

jlebon commented Aug 17, 2020

jlebon commented Sep 28, 2020 •

edited

Loading

jlebon commented Sep 30, 2020

cgwalters left a comment

cgwalters Oct 1, 2020

cgwalters Oct 1, 2020

cgwalters Oct 2, 2020

jlebon Oct 2, 2020 •

edited

Loading

jlebon Oct 2, 2020

cgwalters Oct 2, 2020

cgwalters left a comment

openshift-ci-robot commented Oct 2, 2020

lib/deploy: Add support for overlay initrds #2155

lib/deploy: Add support for overlay initrds #2155

Conversation

jlebon commented Jul 24, 2020 • edited Loading

cgwalters commented Jul 27, 2020

jlebon commented Aug 13, 2020 • edited Loading

jlebon commented Aug 17, 2020

jlebon commented Sep 28, 2020 • edited Loading

jlebon commented Sep 30, 2020

cgwalters left a comment

Choose a reason for hiding this comment

cgwalters Oct 1, 2020

Choose a reason for hiding this comment

cgwalters Oct 1, 2020

Choose a reason for hiding this comment

cgwalters Oct 2, 2020

Choose a reason for hiding this comment

jlebon Oct 2, 2020 • edited Loading

Choose a reason for hiding this comment

jlebon Oct 2, 2020

Choose a reason for hiding this comment

cgwalters Oct 2, 2020

Choose a reason for hiding this comment

cgwalters left a comment

Choose a reason for hiding this comment

openshift-ci-robot commented Oct 2, 2020

jlebon commented Jul 24, 2020 •

edited

Loading

jlebon commented Aug 13, 2020 •

edited

Loading

jlebon commented Sep 28, 2020 •

edited

Loading

jlebon Oct 2, 2020 •

edited

Loading