-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
idea: "upper layer" (erofs) inside of initramfs #332
Comments
One note about performance/memory trade-offs: having the erofs as part of the UKI (and then permanently stored in RAM) would mean that the entire metadata of the system partition is in RAM. |
Another requirement of the "UKI inside the OCI container" approach (and maybe the "UKI generated locally" approach as well): we'd probably want a tool that could scan the UKI to find out which blobs its refers to in the digest store. This is important for pruning the store when removing old images. |
One part of implementing this idea is to adapt https://github.com/ostreedev/ostree/blob/main/src/switchroot/ostree-prepare-root.c to use this EROFS instead of looking at the sysroot. |
Here is a potential flow where we could use that feature that would help us workaround SELinux issues and remove the need for build time commits:
We tried something similar while prototyping: https://github.com/travier/fedora-coreos-uki/blob/main/fcos-uki/Containerfile |
The major change with this approach is that we clearly split the file content from the metadata and the container becomes a way to only transport object data plus a UKI which includes all the metadata. Thus the deployed rootfs becomes an object store only and we don't "care" about ostree commits anymore as we don't need to sign them or use them to regenerate the composefs metadata on the systems. |
Yes. Combining with this comment in general it argues for some new tooling - not too large or complex tooling but new tooling nevertheless. One option is to implement it in this repo as a build-time option - a variant of that is to implement it in Rust (also in this repo). Maybe something like a composefs-boot crate? |
I chatted with @allisonkarlitskaya about this and there's a lot to like about the simplicity of this approach - I'm 100% on board with continuing investigation of this direction. My biggest concern was that I'd also really like to build the story of using composefs for apps/extensions/configmaps etc. and this model reduces the alignment between those two approaches. Combining, this issue also intersects strongly with #294 where I was trying hard to think of a way to bring OCI metadata under verity protection. Hmmm...I guess probably the simplest variant that would work for this is to require the UKI to always be in a distinct layer (with a special annotation like |
In #332 (comment), I forgot that we still need to do the 3-way merge for /etc so we still a "deployment" of it, so this is a bit more complex. |
We've also realized that including the composefs EROFS file in the UKI means that it is now public, thus the the file listing and metadata is public. This is not really an issue but just something to be aware of. |
(when using LUKS on the rootfs) |
For ostree yes, though we also support I think in theory we could ship initramfs glue in this project such that "mount composefs from initramfs" logic could in theory be very agnostic, i.e. we have:
|
Instead of "public" I would say "not encrypted on disk" to be clear. "public" often implies to me "accessible to the whole Internet" but for images generated on premise and deployed to servers that are physically secured, I wouldn't say the UKIs here are "public". That said...AFAIK there's nothing that would block someone from encrypting the erofs in the initramfs, and decrypting using e.g. a key stored in the TPM or something. |
This is a really vague idea that I discussed with @cgwalters and @travier today. They both said it belongs here as an issue. At this point this is little more than a raw braindump. There's a lot to think through and discuss.
The erofs produced by mkcomposefs on a reasonably complete
/usr
is on the order of double digits MB. I've seen ~50MB generally, and it compresses well (down to more like 10MB). The initramfs+kernel on my Silverblue system is low triple digits (~150MB, most of which is the initramfs).It wouldn't be completely unreasonable, then, to have a complete static copy of the composefs "upper layer" erofs image inside of the UKI. This would completely side-step quite a lot of thorny issues around binding the UKI to the correct deployment: all you'd need is the kernel image and the digest store.
How we get a UKI with this erofs inside of it could go two ways:
generate this on the end-user system by (deterministic magic) which lets us get a UKI which is bit-for-bit the same as the one we were expecting it to be. We'd have some out-of-band signature somewhere (in some metadata that doesn't become part of the image) that we could then use for signing this.
push everything to the container image creation: the kernel image would be created as the last step of the image creation process. This would involve running mkcomposefs inside of the container, on the contents of the container itself, and embedding the resulting blob into the UKI, which we'd then write to the container image at a well-known path. Any signing that we might want to do as part of creating the image could happen at this point, inside of the image (or in another build stage and copied back into the final image).
The second approach has an extremely simple deployment strategy: just extract the container .tar directly into a composefs digest store (without creating the erofs). The backing store should now contain all of the files that the erofs referred to. Install the kernel image into the EFI ESP and you're done.
The second way seems wonderfully simple until you realize that there are some very serious drawbacks there:
I think the second approach could be extremely nice for specific deployment scenarios, but it's a very different flavour than what has been promised for the "FROM fedora / ADD / RUN / ..." approach to OS customization.
So that takes us back to a reality where we probably want to support the first scenario of building the composefs and assembling the UKI on the end system. That needs a lot of thinking...
This also intersects with the question about what a signature from an OS vendor on a particular kernel image means. Today it's possible to have a signed kernel boot an unsigned root filesystem. Tomorrow we seem to want to go into a direction where there's additional assurances about the root filesystem contents as well, but if it remains possible to continue booting arbitrary root filesystems with a different version of the same kernel, then this promise is a whole lot less meaningful. In fact, the entire "look how easy it is to customize your system!" bootc ideal sort of depends on being able to modify the root filesystem without needing to resign the kernel... @travier mentioned that we can support both scenarios with kernel variations which produce unique PCR measurements, allowing the data partition to be encrypted by a key that will only be available if we boot a "trusted" rootfs. There are some very deep product-level decisions here...
The text was updated successfully, but these errors were encountered: