Split RHCOS into layers #1637

jlebon · 2024-06-07T15:25:03Z

This enhancement describes improvements to the way RHEL CoreOS (RHCOS) is built so that it will better align with image mode for RHEL, all while also providing benefits on the OpenShift side. Currently, RHCOS is built as a single layer that includes both RHEL and OCP content. This enhancement proposes splitting it into three layers. Going from bottom to top:

the (RHEL-versioned) bootc layer (i.e. the base rhel-bootc image shared with image mode for RHEL)
the (RHEL-versioned) CoreOS layer (i.e. coreos-installer, ignition, afterburn, scripts, etc...)
the (OCP-versioned) node layer (i.e. kubelet, cri-o, etc...)

The terms "bootc layer", "CoreOS layer", and "node layer" will be used throughout this enhancement to refer to these.

The details of this enhancement focus on doing the first split: creating the node layer as distinct from the CoreOS layer (which will not yet be rebased on top of a bootc layer). The two changes involved which most affect OCP are:

bootimages will no longer contain OCP components (e.g. kubelet, cri-o, etc...)
the rhel-coreos payload image will be built in Prow/Konflux (as any other)

Tracked at: https://issues.redhat.com/browse/OCPSTRAT-1190

openshift-ci · 2024-06-07T15:25:08Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

jlebon · 2024-06-07T15:48:37Z

enhancements/rhcos/split-rhcos-into-layers.md

+reviewers:
+  - "@patrickdillon, for installer impact"
+  - "@rphillips, for node impact"
+  - "@joepvd, for ART impact"
+  - "@sinnykumari, for MCO impact"
+  - "@LorbusChris, for OKD impact"
+  - "@zaneb, for agent installer impact"
+  - "@sdodson, for overall architecture"
+  - "@cgwalters, for overall architecture"
+approvers:
+  - "@mrunalp"


Apparently, the bot won't automatically tag the folks listed here, so manually doing it: @patrickdillon @rphillips @joepvd @sinnykumari @LorbusChris @zaneb @sdodson @cgwalters.

cgwalters

Awesome work on this!

enhancements/rhcos/split-rhcos-into-layers.md

rphillips · 2024-06-12T16:00:48Z

openshift/kubernetes has a specific workflow where jobs will build a new kubelet to use during the job run. This helps with rebase work and validating new kubernetes versions coming into OpenShift. We should preserve this workflow when migrating to RHCOS layering.

/cc @soltysh

jlebon · 2024-06-12T16:39:18Z

openshift/kubernetes has a specific workflow where jobs will build a new kubelet to use during the job run. This helps with rebase work and validating new kubernetes versions coming into OpenShift. We should preserve this workflow when migrating to RHCOS layering.

/cc @soltysh

I don't expect any issues there. That workflow should keep working as is.

enhancements/rhcos/split-rhcos-into-layers.md

zaneb · 2024-06-24T11:59:26Z

/cc @cybertron @andfasano

soltysh · 2024-06-26T12:04:05Z

I don't expect any issues there. That workflow should keep working as is.

I believe this was the pre-req work done in openshift/kubernetes#1805, which ensured we won't have problems in o/k.

enhancements/rhcos/split-rhcos-into-layers.md

As per openshift/enhancements#1637, we're trying to get rid of all OpenShift-versioned components from the bootimages. This means that there will no longer be `oc`, `kubelet`, or `crio` binaries for example, which bootstrapping obviously relies on. Instead, now we change things up so that early on when booting the bootstrap node, we pull down the node image, unencapsulate it (this just means convert it back to an OSTree commit), then mount over its `/usr`, and import new `/etc` content. This is done by isolating to a different systemd target to only bring up the minimum number of services to do the pivot and then carry on with bootstrapping. This does not incur additional reboots and should be compatible with AI/ABI/SNO. But it is of course, a huge conceptual shift in how bootstrapping works. With this, we would now always be sure that we're using the same binaries as the target version as part of bootstrapping, which should alleviate some issues such as AI late-binding (see e.g. https://issues.redhat.com/browse/MGMT-16705). The big exception of course being the kernel. Relatedly, currently `/usr/lib/modules` is also shadowed by the mount, but we could re-mount it if needed. To be conservative, the new logic only kicks in when using bootimages which do not have `oc`. This will allow us to ratchet this in more easily. Down the line, we should be able to replace some of this with `bootc apply-live` once that's available (and also works in a live environment). (See containers/bootc#76.) For full context, see the linked enhancement and discussions there.

jlebon · 2024-07-16T21:34:59Z

OK, so let's resume the bootstrapping issue. Restating some of the things from above and from researching further:

We can't run the kubelet in a container because it's no longer supported.
The delta between kubelet and podman play is too large to make the latter a feasible replacement.
systemctl soft-reboot is not in RHEL9.
In the AI/ABI/SNO cases, bootstrapping happens in the live environment where e.g. rebooting is not possible.
I considered cobbling something around kexec, but in the limit, there are potential issues with kexec and hardware reliability, as well as how it meshes with Secure Boot.

What I'm playing with now is basically to have a special node-image-pivot.target that the node isolates to first. There, we pull the node image, unencapsulate it, check out its contents, and then mount over /usr and do a rough 3-way /etc merge. We then isolate back to multi-user.target to continue with the bootstrapping process.

This is in effect like a more aggressive bootc/rpm-ostree apply-live, though that doesn't currently work in live environments. (Though even in the non-live case, there are some issues there that would need to be resolved.) It's close to what OKD currently does when using FCOS live media today, though using the ostree stack and isolating targets should make this more robust.

WIP for this in openshift/installer#8742.

rphillips · 2024-07-16T22:32:52Z

@jlebon That sounds like it might work. Where will the Kubelet be coming from? An OpenShift built image?

zaneb · 2024-07-17T03:52:23Z

Won't doing systemctl isolate node-image-pivot.targethave the effect of stopping the assisted/agent services that we need to avoid stopping?

jlebon · 2024-07-17T14:40:22Z

@jlebon That sounds like it might work. Where will the Kubelet be coming from? An OpenShift built image?

From the node image (i.e. for OCP, the rhel-coreos image in the release payload).

Won't doing systemctl isolate node-image-pivot.targethave the effect of stopping the assisted/agent services that we need to avoid stopping?

No. The system boots into node-image-pivot.target first. Any other service hooked into multi-user.target aren't started until after we've finished the live pivot.

cgwalters · 2024-07-17T14:50:32Z

The system boots into node-image-pivot.target first.

Via a generator overriding default.target?

jlebon · 2024-07-18T21:10:53Z

so we'd need to also add this into the agent ISO and assisted Discovery ISO

A clarification on this: if we can split out those units somehow so they're accessible in the release payload and not baked into the installer, would whatever generates the ISOs be able to pull from the payload so we don't have to duplicate them across codebases? That'd make the more vanilla (non-AI/ABI) install flows more awkward though with a level of indirection.

zaneb · 2024-07-19T00:27:07Z

if we can split out those units somehow so they're accessible in the release payload and not baked into the installer, would whatever generates the ISOs be able to pull from the payload so we don't have to duplicate them across codebases?

We try to avoid having a hard dependency on the release payload in ABI, because the ISO may be generated in a different environment from where the cluster is going to run.

I think it could work to group all of the agent services under e.g. agent.target and in that target set WantedBy: multi-user.target node-image-pivot.target. That way I think we could isolate without stopping those services. Assuming it's safe to do the pivot while stuff is running.

jlebon · 2024-07-19T21:20:16Z

I think it could work to group all of the agent services under e.g. agent.target and in that target set WantedBy: multi-user.target node-image-pivot.target. That way I think we could isolate without stopping those services. Assuming it's safe to do the pivot while stuff is running.

Yeah, that's along the lines of what I was thinking as well. I like the agent.target idea.

I might reach out with questions for getting a test environment up to iterate on this.

enhancements/rhcos/split-rhcos-into-layers.md

As per openshift/enhancements#1637, we're trying to get rid of all OpenShift-versioned components from the bootimages. This means that there will no longer be `oc`, `kubelet`, or `crio` binaries for example, which bootstrapping obviously relies on. Instead, now we change things up so that early on when booting the bootstrap node, we pull down the node image, unencapsulate it (this just means convert it back to an OSTree commit), then mount over its `/usr`, and import new `/etc` content. This is done by isolating to a different systemd target to only bring up the minimum number of services to do the pivot and then carry on with bootstrapping. This does not incur additional reboots and should be compatible with AI/ABI/SNO. But it is of course, a huge conceptual shift in how bootstrapping works. With this, we would now always be sure that we're using the same binaries as the target version as part of bootstrapping, which should alleviate some issues such as AI late-binding (see e.g. https://issues.redhat.com/browse/MGMT-16705). The big exception of course being the kernel. Relatedly, note we do persist `/usr/lib/modules` from the booted system so that loading kernel modules still works. To be conservative, the new logic only kicks in when using bootimages which do not have `oc`. This will allow us to ratchet this in more easily. Down the line, we should be able to replace some of this with `bootc apply-live` once that's available (and also works in a live environment). (See containers/bootc#76.) For full context, see the linked enhancement and discussions there.

As per openshift/enhancements#1637, we're trying to get rid of all OpenShift-versioned components from the bootimages. This means that there will no longer be oc, kubelet, or crio binaries for example, which bootstrapping obviously relies on. To adapt to this, the OpenShift installer now ships a new `node-image-overlay.service` in its bootstrap Ignition config. This service takes care of pulling down the node image and overlaying it, effectively updating the system to the node image version. Here, we accordingly also adapt assisted-installer so that we run `node-image-overlay.service` before starting e.g. `kubelet.service` and `bootkube.service`. See also: openshift/installer#8742

As per openshift/enhancements#1637, we're trying to get rid of all OpenShift-versioned components from the bootimages. This means that there will no longer be `oc`, `kubelet`, or `crio` binaries for example, which bootstrapping obviously relies on. Instead, now we change things up so that early on when booting the bootstrap node, we pull down the node image, unencapsulate it (this just means convert it back to an OSTree commit), then mount over its `/usr`, and import new `/etc` content. This is done by isolating to a different systemd target to only bring up the minimum number of services to do the pivot and then carry on with bootstrapping. This does not incur additional reboots and should be compatible with AI/ABI/SNO. But it is of course, a huge conceptual shift in how bootstrapping works. With this, we would now always be sure that we're using the same binaries as the target version as part of bootstrapping, which should alleviate some issues such as AI late-binding (see e.g. https://issues.redhat.com/browse/MGMT-16705). The big exception of course being the kernel. Relatedly, note we do persist `/usr/lib/modules` from the booted system so that loading kernel modules still works. To be conservative, the new logic only kicks in when using bootimages which do not have `oc`. This will allow us to ratchet this in more easily. Down the line, we should be able to replace some of this with `bootc apply-live` once that's available (and also works in a live environment). (See containers/bootc#76.) For full context, see the linked enhancement and discussions there.

openshift-ci · 2024-09-09T20:14:27Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign miciah for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jlebon · 2024-09-09T20:17:42Z

Updated and officially ready for review!

Filled in the remaining sections
Added details about versioning scheme
Added details about solving the bootstrapping issue
Added @yuqi-zhang and @jupierce as reviewers

cgwalters · 2024-09-10T12:49:59Z

@omertuc did you see this one?

As per openshift/enhancements#1637, we're trying to get rid of all OpenShift-versioned components from the bootimages. This means that there will no longer be oc, kubelet, or crio binaries for example, which bootstrapping obviously relies on. To adapt to this, the OpenShift installer now ships a new `node-image-overlay.service` in its bootstrap Ignition config. This service takes care of pulling down the node image and overlaying it, effectively updating the system to the node image version. Here, we accordingly also adapt assisted-installer so that we run `node-image-overlay.service` before starting e.g. `kubelet.service` and `bootkube.service`. See also: openshift/installer#8742

As per openshift/enhancements#1637, we're trying to get rid of all OpenShift-versioned components from the bootimages. This means that there will no longer be `oc`, `kubelet`, or `crio` binaries for example, which bootstrapping obviously relies on. Instead, now we change things up so that early on when booting the bootstrap node, we pull down the node image, unencapsulate it (this just means convert it back to an OSTree commit), then mount over its `/usr`, and import new `/etc` content. This is done by isolating to a different systemd target to only bring up the minimum number of services to do the pivot and then carry on with bootstrapping. This does not incur additional reboots and should be compatible with AI/ABI/SNO. But it is of course, a huge conceptual shift in how bootstrapping works. With this, we would now always be sure that we're using the same binaries as the target version as part of bootstrapping, which should alleviate some issues such as AI late-binding (see e.g. https://issues.redhat.com/browse/MGMT-16705). The big exception of course being the kernel. Relatedly, note we do persist `/usr/lib/modules` from the booted system so that loading kernel modules still works. To be conservative, the new logic only kicks in when using bootimages which do not have `oc`. This will allow us to ratchet this in more easily. Down the line, we should be able to replace some of this with `bootc apply-live` once that's available (and also works in a live environment). (See containers/bootc#76.) For full context, see the linked enhancement and discussions there.

enhancements/rhcos/split-rhcos-into-layers.md

jupierce · 2024-09-11T23:31:23Z

enhancements/rhcos/split-rhcos-into-layers.md

+container image, just like most other OpenShift components are built. This should
+allow simplifying CI and ART tooling related to RHCOS, much of which is bespoke.
+
+For example, ART would no longer have to sync RHCOS images to CI; CI could


I believe this would still occur. CI builds / rebuilds images after a change merges. Images in repos that don't have frequent merges (e.g. openshift/images) go stale over time -- which is why pushes updates it builds for these images into the CI integration imagestreams.
For ART to stop doing this, we'd need to know (a) what triggers the RHCOS build in CI (b) what keeps it fresh (c) how it could stay current with RHEL & OCP RPM updates.
The statements which follow remain accurate even if ART continues to build the image.

Thanks, that's a detail I didn't know but it makes sense. So overall images in the CI imagestreams may come from either CI or ART nightlies depending on how fresh the repo is?

I'm trying to verify my understanding of this by comparing the image digests in e.g. registry.ci.openshift.org/ocp/release:4.17.0-0.ci-2024-09-24-105044 vs registry.ci.openshift.org/ocp/release:4.17.0-0.nightly-2024-09-23-211417; I would've expected some of them to be the same there. The CI payloads are manifest listed but not the nightly ART one, but even comparing against the amd64 digests in the manifest lists against the nightly ones, I'm not getting any component images that match other than the RHCOS ones.

Hmm, and based on this other comment, are you suggesting not having CI builds promote at all and keep relying on just ART or both? I'd like the CoreOS images to be at parity with the other component images. E.g. merges to openshift/os would ideally trigger a CI build and promotion.

enhancements/rhcos/split-rhcos-into-layers.md

jupierce · 2024-09-11T23:54:27Z

enhancements/rhcos/split-rhcos-into-layers.md

+which will build the OpenShift node image by layering the OpenShift components on top: RPM
+packages, config files, etc...
+
+After some testing, the Prow job will be configured to push to the `rhel-coreos` tag in the CI


I think this is a perfect option to enable effective pre-merge CI for openshift/os, but as discussed above, I'm not sure it should stop ART promotion of the rhcos content to the CI imagestream.
Consider an older z-stream like 4.12. Very few merges to openshift/os in release-4.12 would mean, in effect, no updates the rhcos content in the 4.12 CI imagestream.
An update merging in openshift/oc would not update the client RPM in the CI coreos image until (a) ART rebuilt the client RPM and (b) a subsequent PR merged in openshift/os causing that new RPM to be installed.

Agreed. Will update this section to make it match what we want after the conversation above about this is resolved.

enhancements/rhcos/split-rhcos-into-layers.md

jupierce · 2024-09-12T00:22:24Z

enhancements/rhcos/split-rhcos-into-layers.md

+    Brew and those builds then being added to the ART plashets. To avoid having
+    to source the (OCP-versioned) ART plashets during the CoreOS builds, we
+    could instead create new repos for these packages that are clearly defined
+    to be RHEL-versioned. The CoreOS build could then add that repo alongside


I think ART would need to dnf upgrade the kernel in the node layer from their plashet. If the coreos layer is up-to-date, it is a no-op. Otherwise, we lose some of the benefit of the layer decoupling.

Hmm, when would it be not up to date? The ART plashet sources it from the same place the RHCOS image would source it from. It's certainly possible to replace the kernel in the node image build, but IMO it's much more appropriate to respin the RHCOS base image for it.

enhancements/rhcos/split-rhcos-into-layers.md

This enhancement describes improvements to the way RHEL CoreOS (RHCOS) is built so that it will better align with image mode for RHEL, all while also providing benefits on the OpenShift side. Currently, RHCOS is built as a single layer that includes both RHEL and OCP content. This enhancement proposes splitting it into three layers. Going from bottom to top: 1. the (RHEL-versioned) bootc layer (i.e. the base rhel-bootc image shared with image mode for RHEL) 2. the (RHEL-versioned) CoreOS layer (i.e. coreos-installer, ignition, afterburn, scripts, etc...) 3. the (OCP-versioned) node layer (i.e. kubelet, cri-o, etc...) The terms "bootc layer", "CoreOS layer", and "node layer" will be used throughout this enhancement to refer to these. The details of this enhancement focus on doing the first split: creating the node layer as distinct from the CoreOS layer (which will not yet be rebased on top of a bootc layer). The two changes involved which most affect OCP are: 1. bootimages will no longer contain OCP components (e.g. kubelet, cri-o, etc...) 2. the `rhel-coreos` payload image will be built in Prow/Konflux (as any other) Tracked at: https://issues.redhat.com/browse/OCPSTRAT-1190

openshift-ci · 2024-09-24T19:00:49Z

@jlebon: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 7, 2024

jlebon mentioned this pull request Jun 7, 2024

Rework build process to generate rhel-coreos-base distinct from ocp-rhel-coreos openshift/os#799

Closed

jlebon force-pushed the pr/split-rhcos-into-layers branch from 067ece5 to f79684b Compare June 7, 2024 15:38

jlebon commented Jun 7, 2024

View reviewed changes

cgwalters reviewed Jun 7, 2024

View reviewed changes

travier reviewed Jun 10, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

travier reviewed Jun 10, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

sdodson reviewed Jun 10, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved

openshift-ci bot requested a review from soltysh June 12, 2024 16:00

jlebon force-pushed the pr/split-rhcos-into-layers branch from f79684b to a6a7438 Compare June 20, 2024 21:15

zaneb reviewed Jun 24, 2024

View reviewed changes

openshift-ci bot requested review from andfasano and cybertron June 24, 2024 11:59

rphillips reviewed Jun 26, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved

romfreiman reviewed Jun 30, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

romfreiman reviewed Jun 30, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

romfreiman reviewed Jun 30, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

andfasano reviewed Jul 11, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Outdated Show resolved Hide resolved

jlebon mentioned this pull request Jul 16, 2024

WIP: overlay node image before bootstrapping if necessary openshift/installer#8742

Draft

yuqi-zhang reviewed Jul 21, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

yuqi-zhang reviewed Jul 21, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

cverna reviewed Aug 21, 2024

View reviewed changes

enhancements/rhcos/split-rhcos-into-layers.md Show resolved Hide resolved

jlebon mentioned this pull request Aug 22, 2024

Run a basic cluster test in Prow openshift/os#1584

Open

jlebon mentioned this pull request Sep 6, 2024

WIP: overlay node image before bootstrapping if necessary openshift/assisted-installer#899

Draft

jlebon force-pushed the pr/split-rhcos-into-layers branch from a6a7438 to 4e54497 Compare September 9, 2024 20:14

jlebon marked this pull request as ready for review September 9, 2024 20:16

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 9, 2024

openshift-ci bot requested review from jaypoulz and shawn-hurley September 9, 2024 20:16

jlebon force-pushed the pr/split-rhcos-into-layers branch from 4e54497 to 086ae67 Compare September 9, 2024 23:30

jupierce reviewed Sep 12, 2024

View reviewed changes

jlebon force-pushed the pr/split-rhcos-into-layers branch from 086ae67 to 02e69db Compare September 24, 2024 18:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split RHCOS into layers #1637

Split RHCOS into layers #1637

jlebon commented Jun 7, 2024

openshift-ci bot commented Jun 7, 2024

jlebon Jun 7, 2024

cgwalters left a comment

rphillips commented Jun 12, 2024

jlebon commented Jun 12, 2024

zaneb commented Jun 24, 2024

soltysh commented Jun 26, 2024

jlebon commented Jul 16, 2024

rphillips commented Jul 16, 2024

zaneb commented Jul 17, 2024

jlebon commented Jul 17, 2024

cgwalters commented Jul 17, 2024

jlebon commented Jul 18, 2024

zaneb commented Jul 19, 2024

jlebon commented Jul 19, 2024

openshift-ci bot commented Sep 9, 2024

jlebon commented Sep 9, 2024

cgwalters commented Sep 10, 2024

jupierce Sep 11, 2024

jlebon Sep 24, 2024

jupierce Sep 11, 2024

jlebon Sep 24, 2024

jupierce Sep 12, 2024

jlebon Sep 24, 2024

openshift-ci bot commented Sep 24, 2024

Split RHCOS into layers #1637

Are you sure you want to change the base?

Split RHCOS into layers #1637

Conversation

jlebon commented Jun 7, 2024

openshift-ci bot commented Jun 7, 2024

jlebon Jun 7, 2024

Choose a reason for hiding this comment

cgwalters left a comment

Choose a reason for hiding this comment

rphillips commented Jun 12, 2024

jlebon commented Jun 12, 2024

zaneb commented Jun 24, 2024

soltysh commented Jun 26, 2024

jlebon commented Jul 16, 2024

rphillips commented Jul 16, 2024

zaneb commented Jul 17, 2024

jlebon commented Jul 17, 2024

cgwalters commented Jul 17, 2024

jlebon commented Jul 18, 2024

zaneb commented Jul 19, 2024

jlebon commented Jul 19, 2024

openshift-ci bot commented Sep 9, 2024

jlebon commented Sep 9, 2024

cgwalters commented Sep 10, 2024

jupierce Sep 11, 2024

Choose a reason for hiding this comment

jlebon Sep 24, 2024

Choose a reason for hiding this comment

jupierce Sep 11, 2024

Choose a reason for hiding this comment

jlebon Sep 24, 2024

Choose a reason for hiding this comment

jupierce Sep 12, 2024

Choose a reason for hiding this comment

jlebon Sep 24, 2024

Choose a reason for hiding this comment

openshift-ci bot commented Sep 24, 2024