-
Notifications
You must be signed in to change notification settings - Fork 158
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
40ignition-ostree: add coreos-inject-rootmap.service #503
Conversation
Requires #184, #354, and the SELinux patch.
Example Ignition config: {
"ignition": {
"version": "3.2.0-experimental"
},
"storage": {
"disks": [
{
"device": "/dev/disk/by-id/virtio-disk1",
"partitions": [
{
"label": "foo"
}
],
"wipeTable": true
},
{
"device": "/dev/disk/by-id/virtio-disk2",
"partitions": [
{
"label": "bar"
}
],
"wipeTable": true
}
],
"raid": [
{
"devices": [
"/dev/disk/by-partlabel/foo",
"/dev/disk/by-partlabel/bar"
],
"level": "raid1",
"name": "myroot"
}
],
"filesystems": [
{
"device": "/dev/md/myroot",
"format": "xfs",
"wipeFilesystem": true,
"label": "root"
}
]
}
} Running with cosa:
|
For fun, with coreos/ignition#1010, it's possible to do root on RAID10: {
"ignition": {
"version": "3.2.0-experimental"
},
"storage": {
"disks": [
{
"device": "/dev/disk/by-id/virtio-disk1",
"partitions": [
{
"label": "foo"
}
],
"wipeTable": true
},
{
"device": "/dev/disk/by-id/virtio-disk2",
"partitions": [
{
"label": "bar"
}
],
"wipeTable": true
},
{
"device": "/dev/disk/by-id/virtio-disk3",
"partitions": [
{
"label": "baz"
}
],
"wipeTable": true
},
{
"device": "/dev/disk/by-id/virtio-disk4",
"partitions": [
{
"label": "boo"
}
],
"wipeTable": true
}
],
"raid": [
{
"devices": [
"/dev/disk/by-partlabel/foo",
"/dev/disk/by-partlabel/bar"
],
"level": "raid1",
"name": "foobar"
},
{
"devices": [
"/dev/disk/by-partlabel/baz",
"/dev/disk/by-partlabel/boo"
],
"level": "raid1",
"name": "bazboo"
},
{
"devices": [
"/dev/md/foobar",
"/dev/md/bazboo"
],
"level": "raid0",
"name": "myroot"
}
],
"filesystems": [
{
"device": "/dev/md/myroot",
"format": "xfs",
"wipeFilesystem": true,
"label": "root"
}
]
}
} Running with cosa:
|
So... I initially wanted to write this in rpm-ostree so it can be in not-bash, and then call it from the initramfs, but it's hard to beat bash in prototyping speed. And then it ended up being much simpler than expected thanks to |
overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-inject-rootmap.service
Outdated
Show resolved
Hide resolved
overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-uuid-boot.service
Show resolved
Hide resolved
overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-dracut-rootfs.sh
Outdated
Show resolved
Hide resolved
overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-inject-rootmap.service
Outdated
Show resolved
Hide resolved
overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-inject-rootmap.service
Outdated
Show resolved
Hide resolved
overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-inject-rootmap
Outdated
Show resolved
Hide resolved
overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-inject-rootmap
Outdated
Show resolved
Hide resolved
overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-inject-rootmap
Outdated
Show resolved
Hide resolved
Re. root-on-LUKS, see coreos/ignition#960 (comment). |
OK, rebased this and now with #466, #184, and #354 folded in! Will see how quickly we can get a respin of Clevis with latchset/clevis#211 included. Otherwise, we should be able to hack around it for now by carrying our own version of the path and service units until that happens. I haven't addressed the comments yet. I'd also like to write a kola test for this before everything goes in. Also need to adapt growpart. |
overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-growpart
Outdated
Show resolved
Hide resolved
...ay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-rootfs-detect.service
Show resolved
Hide resolved
This is blocked on https://bugzilla.redhat.com/show_bug.cgi?id=1845210. |
I checked out this branch and also dropped in a locally built kernel from this commit, then provided:
And it just worked, rootfs on btrfs was that easy! (And same for (I did get a failure from |
Another bug I just hit with this is when doing an install without providing an Ignition config:
|
The `platform.Conf` type allows abstracting over the different Ignition versions so that different tests can use different versions. Using it instead of the Ignition type directly means that we can now use the 3.2-experimental spec in external tests. This is needed for testing the new LUKS support in Ignition[1] and the related rootfs-on-complex-devices work[2]. [1] coreos/ignition#960 [2] coreos/fedora-coreos-config#503
The `platform.Conf` type allows abstracting over the different Ignition versions so that different tests can use different versions. Using it instead of the Ignition type directly means that we can now use the 3.2-experimental spec in external tests. This is needed for testing the new LUKS support in Ignition[1] and the related rootfs-on-complex-devices work[2]. [1] coreos/ignition#960 [2] coreos/fedora-coreos-config#503 Closes: coreos#1589
The `platform.Conf` type allows abstracting over the different Ignition versions so that different tests can use different versions. Using it instead of the Ignition type directly means that we can now use the 3.2-experimental spec in external tests. This is needed for testing the new LUKS support in Ignition[1] and the related rootfs-on-complex-devices work[2]. [1] coreos/ignition#960 [2] coreos/fedora-coreos-config#503 Closes: coreos#1589
At least for the no such file part with the new Ignition release the cache config ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks good at a high level!
overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-growpart
Show resolved
Hide resolved
overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-growpart
Show resolved
Hide resolved
overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-dracut-rootfs.sh
Outdated
Show resolved
Hide resolved
OK, so one thing I just realized is that with (Edit: sorry I was kinda confused on this. Mount options for root can't live in the Ignition config. So in the reprovisioning case, it's up to users to add an appropriate |
This adds basic infrastructure units for "re-provisioning" the root filesystem. See: coreos/fedora-coreos-tracker#94 A unit first detects if the Ignition configuration has a filesystem with the label `root` - if so, we save the rootfs into RAM, let `ignition-disks.service` run, then restore it from RAM. Earlier attempts used the `brd` kernel module which is a RAM-backed block device so we can just `dd`. However, this has some limitations, such as the need to save the full disk in RAM, and the inability for any other initrd code to use `brd` devices. As well, `brd` doesn't support discards, so we require at minimum $rootfs_size RAM (e.g. 3G) until reprovisioning is complete. Future work here will likely move the `restore` phase into `rpm-ostree`. Co-authored-by: Jonathan Lebon <jonathan@jlebon.com>
This is a general best practice; the intention of filesystem UUIDs is that they're unique. It helps backup systems and the like if we change this. This builds on coreos/coreos-assembler@e3905fd In the future, we may also switch to using these UUIDs for subsequent boots; see: coreos/fedora-coreos-tracker#465
This implements the rootmap functionality that figures out all the dependencies required to find `/sysroot`, and injects them into the BLS config. For more information, see: coreos/fedora-coreos-tracker#94 (comment) The `rdcore` code supports RAID and LUKS devices, though the latter needs a new Clevis release with the following patches to be fully supported: latchset/clevis#211 latchset/clevis#217 This also implements the `root=UUID=$uuid` inject patch proposed in coreos/fedora-coreos-tracker#465. On its own, this unlocks reprovisioning FCOS with root on a RAID device, or e.g. in-place reprovisioning of root on btrfs. Closes: coreos/fedora-coreos-tracker#465 Closes: coreos/fedora-coreos-tracker#94
This stamp file was used to make sure coreos-growpart only ran on the first boot when it ran in the real root. Nowadays, it runs in the initramfs as part of `ignition-complete.target` before we even run `ostree-prepare-root` (which meant we were actually creating the stamp file in the initrd filesystem). Having a stamp file is useful though for writing tests. So let's repurpose the idea and put it in `/run` instead.
Add two basic tests: one where we reprovision in place to ext4, and one where we reprovision onto a separate RAID1.
That generator no longer exists.
Split out a small script where the canonical rootflags live in the non-reprovisioning case. This will be used by both `ignition-ostree-mount-sysroot` and `rdcore rootmap`.
This is fixed now in coreos/coreos-installer#358 and the latest commit here which adds |
🎉 Who wants to give an approving review on this so we can merge it in? |
fi | ||
mount -o "${mountflags}" "${rootpath}" /sysroot | ||
|
||
mount -o "$(coreos-rootflags)" "${rootpath}" /sysroot |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This took a second to realize it was sensing options based on a script but 👍!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work here! I seriously appreciate the additional commenting through the code 🙏
@jlebon Thanks for making this happen! 🎉 🙏 |
Hello, |
Hmm, I think if we have magic for reprovisioning root, then we should also just handle |
This implements the rootmap functionality that figures out all the
dependencies required to find
/sysroot
, and injects them into the BLSconfig. For more information, see:
coreos/fedora-coreos-tracker#94 (comment)
The
rdcore
code supports RAID and LUKS devices, though the latterneeds a new Clevis release with the following patches to be fully
supported:
latchset/clevis#211
latchset/clevis#217
This also implements the
root=UUID=$uuid
inject patch proposed incoreos/fedora-coreos-tracker#465.
On its own, this unlocks reprovisioning FCOS with root on a RAID device,
or e.g. in-place reprovisioning of root on btrfs.
Closes: coreos/fedora-coreos-tracker#465
Closes: coreos/fedora-coreos-tracker#94