Support disabling initramfs networking via Ignition #979
Comments
|
@cgwalters What do you think of something like this? |
|
To link this to a real (OpenShift) use-case, see https://bugzilla.redhat.com/show_bug.cgi?id=1824331 - in some baremetal environments the external network can be on a tagged vlan, so you have to do some configuration via ignition before the |
|
A lot of overlap here with openshift/enhancements#291 |
I did read through that but it's very tied to solving the problem by embedding an ignition in an ISO, rather than letting ignition do some things before and after networkmanger is up. |
Yea that's a similar problem, but we are using https://coreos.com/os/docs/latest/config-drive.html instead, so I'm not sure this issue helps us |
|
What I'd argued in one of the epic number of tickets around this is that I think all we need to do for all the major clouds that use the link-local address is bring up networking enough to do that, then we could pull network configuration out of the Ignition itself - then we can do static networking in clouds too (not that most people want this). That said, if we fixed openshift/machine-config-operator#1690 then you guys could "flatten" the config into
But we'd need to flesh out more what that would look like. Something like: Or just support writing arbitrary things into |
That addresses the immediate problem for us, but not the general use. It makes complete sense to use network data sources for append/merge in ignition, but there's always going to be a circular dependency on environments that have networks that need configuration.
Maybe? I don't understand entirely how the initramfs dracut/ignition stuff interact -- is the /etc/NetworkManager at that stage in the ramdisk or is that the host's eventual configuration? |
|
We may need a meeting on this but I really want to walk though how the Live ISO changes the game for you guys. Hmm....I think we maybe do need to support some flag/mechanism for the Live ISO to disable networking in the initramfs even if Ignition is injected.
We use NM in the initrd too (RHCOS will after 8.2). There's no required linkage between the initramfs configuration and the real root; most people will want them to be the same (see coreos/ignition-dracut#89 ) but like that PR we can add an option to avoid propagating them to the real root. |
|
Related: #956 (which would also be smart enough to handle live ISO with embedded Ignition configs). It's also an on-ramp for supporting fetching Ignition configs over link-local addresses. |
|
I'm very hesitant to allow Ignition to configure the initramfs itself. I'd prefer we stay away from that for as long as we can because it essentially introduces the same problem for the initrd which Ignition was meant to solve for the real root.
We've been working hard on improving the situation there. :) In addition to the links above, see also coreos/fedora-coreos-tracker#460. I agree with Colin that we should push further on the live ISO embedded config path for bare metal, and investigate the link-local approach for clouds. (I'm also not sure how Ignition can correctly implement the RFE here if e.g. you configure networking in the base config, but then one of the |
Would it be kernel command line arguments then for this configuration? I wasn't sure if that worked in rhcos, since we tried it when we were trying to get IPv6 working. Can you do VLAN configuration with those arguments?
The ISO approach for bare metal isn't going to work in the IPI case, and that seems to be ignored in the present designs. We do exist and have customers. We have some initial discussions about how we can use the live ISO for IPI but it's going to be later than we need this problem solved. I also don't see how the live ISO case fits into OpenShift, because the installer generates a stub ignition with network dependencies and you're going to run into customers with these complex network requirements. I don't see where that use case is being addressed.
You could have something like afterburn to be able to do preflight configuration before ignition runs. OpenStack config drives have network metadata for similar reasons: https://docs.openstack.org/nova/latest/user/metadata.html. |
If it's just constrained to networking though, that seems not too bad. And actually perhaps to start, we could offer exactly one thing - the ability for Ignition to disable initramfs networking. That would allow using In particular once we have osmet, I suspect a lot of cases could get away with not even bringing up the network at all in the live OS. |
cgwalters
changed the title
|
Yeah I am increasingly certain that we missed this important case of "Ignition provided, but does not require networking in the initramfs" in all of the recent work following coreos/fedora-coreos-tracker#443 I took a look at the recent libvirt testing scripts and it looks like to me theyl use the default bridged network which still has DHCP enabled. So what happens here is when we're doing a static config in those tests...we still used (and required!) DHCP in the initramfs. That's not going to work for environments without DHCP at all, or even just ones where we don't want to DHCP on each NIC but not all of them etc. openshift/enhancements#291 is not complete IMO until we fix this. Oh right actually, this is exactly what this issue is: coreos/coreos-installer#164 |
|
OK so yep, going back we have two "not so complex" options:
Or, the much bigger options of:
Another approach is:
|
A follow-on that we still want to do is coreos/fedora-coreos-tracker#460
Can you elaborate what you mean. That test script tests many different configurations including dhcp and static networking in the initramfs combined with static networking in the real root. I do use the default libvirt bridged network, but that's only to cover the dhcp test cases and the machine won't attempt DHCP if you tell it not to.
The EPIC work that I've done specifically focuses on static networking via the installer in the Live ISO. In that case you can easily configure static networking when doing the install. Follow on work to this is to not bring up networking in the initramfs if not needed at all: coreos/fedora-coreos-tracker#460, but that was not part of the EPIC.
I think the better approach is
but I guess the real question is why do you need "no networking in the initramfs". I think the real desire is "I only want the networking I want in the initramfs". In that case just configure the networking you want via the installer right? |
|
Heh, was typing up a comment but @dustymabe beat me to it. :) To summarize: In the general case, I think this is resolved by coreos/fedora-coreos-tracker#460 (which provides a framework for apps to tell the OS that networking is needed) and #956 + coreos/ignition-dracut#164 (which makes use of that framework). In the live ISO case/metal case of "complex networking on first boot", one can configure networking as required at install time and use |
|
@jlebon yes. Maybe we should all get together and write up a proper EPIC so the requirements are collected and we can make sure coreos/fedora-coreos-tracker#460 addresses all needs. |
The user story here is "I have a CoreOS ISO and |
Right, I added that as a second option in the "more complex" options above. |
Yep. We did have coreos/fedora-coreos-config#326 to not require networking for the ISO but we explicitly do bring up networking in the case of an embedded ignition config (as requested during feature discussions) and pointing to future work to take care of that case: https://github.com/coreos/fedora-coreos-config/blob/5615f58487f9fd9f1ebc19e0e6416b53a3e6270f/overlay.d/05core/usr/lib/dracut/modules.d/20live/coreos-liveiso-network-kargs.service#L5-L15 We could easily remove the |
"but the user can override with No, there's no ergonomic way to override the live ISO kernel arguments in an automated fashion.
That's an option I guess...I added it to the list above. |
Right yeah, this is also in scope for coreos/fedora-coreos-tracker#460. Right now we took the shortcut that embedded config = requires networking. But the end goal is absolutely for Ignition to only ask for networking if required (via the exact same mechanism that is used in clouds/freshly installed metal). One gap then is if one needs complex networking on first boot of a non-metal platform, and that's what Afterburn is trying to address (by providing a channel for configuring networking, see e.g. coreos/afterburn#404). Of course, another major gap for much of this is to have osmet supported on RHCOS so that install-time can truly be fully offline. We should chat about our approach on that! (And I see now you recently opened coreos/coreos-assembler#1467 related to this). |
|
This discussion seems to be going off the rails a bit from our original feature request. I'm not sure why the live CD is part of this particular discussion, we're not using it -- maybe we should have a live discussion about what ignition is missing for us? We need to have something in the 4.6 time frame. OpenShift installer creates an ignition that needs networking. Any ignition has the possibility to include merge/appends that reference network URI's -- what is the answer when a user requires some configuration to make networking work to access those resources? It's impossible today in the baremetal IPI platform to support use cases where the network used to access the MCS requires a VLAN, for example. |
|
From #979 (comment)
Networking would still be enabled in the other conditions. Just not for the embedded ignition config case. |
👍 Yeah, that'd be great! :) The reason we're talking about the live ISO is because it will break the dependency loop between Ignition and networking by being able to provide networking configuration upfront at install time. |
Agreed. The result of this spike should find it's way into a card for execution as well once we've figured out what should be done. |
Yes, I understand. If we want to handle your case of using the
OK I thought about this more and you're right, we either need to:
Of these...I lean more towards the first, but it's hard. The reasons (AFAIK)
Anyways, once we have a flattened config, then as long as we have a way to just disable the initramfs networking requirement, we only have to concern ourselves with networking in the real root. |
Make use of the new `fetch-offline` stage: coreos/ignition#979 We run this between the `setup` and `fetch` stages (the latter possibly being skipped if networking is not required). We hit the same issue here that `coreos-copy-firstboot-network.service` hit, which is that we can't run before the `cmdline` hook because that runs *before* udev, but we want the `by-*` symlinks for `ignition-setup-user.service`. The hack we do here is to rerun the NM cmdline hook in case ignition dropped a snippet in `/etc/cmdline.d`. As mentioned in coreos/fedora-coreos-config#346, we'll be able to do this more cleanly once we run NM as a systemd service directly.
Make use of the new `fetch-offline` stage: coreos/ignition#979 We run this between the `setup` and `fetch` stages (the latter possibly being skipped if networking is not required). We hit the same issue here that `coreos-copy-firstboot-network.service` hit, which is that we can't run before the `cmdline` hook because that runs *before* udev, but we want the `by-*` symlinks for `ignition-setup-user.service`. The hack we do here is to rerun the NM cmdline hook in case ignition dropped a snippet in `/etc/cmdline.d`. As mentioned in coreos/fedora-coreos-config#346, we'll be able to do this more cleanly once we run NM as a systemd service directly.
Make use of the new `fetch-offline` stage: coreos/ignition#979 We run this between the `setup` and `fetch` stages (the latter possibly being skipped if networking is not required). We hit the same issue here that `coreos-copy-firstboot-network.service` hit, which is that we can't run before the `cmdline` hook because that runs *before* udev, but we want the `by-*` symlinks for `ignition-setup-user.service`. The hack we do here is to rerun the NM cmdline hook in case ignition dropped a snippet in `/etc/cmdline.d`. As mentioned in coreos/fedora-coreos-config#346, we'll be able to do this more cleanly once we run NM as a systemd service directly.
Make use of the new `fetch-offline` stage: coreos/ignition#979 We run this between the `setup` and `fetch` stages (the latter possibly being skipped if networking is not required). We hit the same issue here that `coreos-copy-firstboot-network.service` hit, which is that we can't run before the `cmdline` hook because that runs *before* udev, but we want the `by-*` symlinks for `ignition-setup-user.service`. The hack we do here is to rerun the NM cmdline hook in case ignition dropped a snippet in `/etc/cmdline.d`. As mentioned in coreos/fedora-coreos-config#346, we'll be able to do this more cleanly once we run NM as a systemd service directly.
Thanks @jlebon - this is the detail I was missing :) I wonder if we could consider making the 15copy-installer-network script copy from a config-drive partition if it exists? One of the problems when the ironic deploy ramdisk is used (as is currently the case for IPI baremetal) it can write a config-drive partition, but there's no support for modifying the OS disk image itself (so we can't drop files in /boot). Long term we can also look at how we might consume the rhcos initrd with coreos-install, but there are some feature gaps and interfaces differences which will need further investigation if we want that to work with the ironic services we currently use. |
Hmm... maybe. (Though the fact that it's optional makes it similar to #928.) Let's see if there's a better way first.
Do you have a link to documentation/code re. what the deployer is capable of? I presume you're not able to run arbitrary code?
Yes, agreed! 👍 |
Long term I think there are a few options with varying levels of difficulty so we might progressively go through them:
|
There is an overview here https://docs.openstack.org/ironic/latest/user/index.html#understanding-bare-metal-deployment - the agent itself is https://github.com/openstack/ironic-python-agent which is contained in a ramdisk we use to write the OCP disk image (similar to the OCP ramdisk with coreos-install). Basically there are no interfaces for running arbitrary code, or injecting arbitrary files, the baremetal box is treated like a cloud VM - you provide an OS disk image, and optionally some user-data/meta-data (which in the metal3 case happens via a config-drive partition) OpenStack has support for a network_data.json ref https://docs.openstack.org/nova/latest/user/metadata.html#openstack-format-metadata but I don't think the ignition openstack backend does anything with that atm? |
Yeah I suspect that won't be acceptable to upstream Ironic folks as a change to ironic-python-agent, since it's designed to be completely OS agnostic /cc @juliakreger @dtantsur
I suspect this is where we want to get to, the main challenge is right now we depend on some features of ironic-python-agent (in particular introspection), so the RHCOS/FCOS ramdisk isn't a drop-in replacement. Switching to an RHCOS/FCOS based solution would either require reimplementing some of the IPA features (and interfaces to interact with ironic, but that could be a shim injected like in your blog post I guess), or maybe have a way to run IPA on top of the RHCOS/FCOS image (which doesn't solve the inject-files-to-boot problem but it has been done before ref https://docs.openstack.org/ironic-python-agent/queens/install/index.html#coreos) |
Yeah, it would practically require a downstream addition to the ramdisk. I'd like to note that ironic-python-agent is designed to be pretty flexible, so it's not an entirely bad thing either. We're working on a thing called "deploy steps" specifically to allow site-specific customizations.
I think CoreOS was so different at the time that it won't help us (and we did some ugly things like chroot in tarball contents). |
OK right, that makes a lot of sense now.
So to recap, I think we're agreed on where we want to go long-term. And the question now is how to get this working in the short-term with the existing constraints of the deployer. @hardys Do I understand correctly that the "edit image" hack should work for now? If we can do the flattened Ignition config thing, this could also be solved by coreos/fedora-coreos-config#426. |
Sorry for the delay responding @jlebon yes I think that's accurate, so to summarize:
|
In recent ironic (~ a week ago) you can drop a python plugin to IPA that will be called automatically during deployment to do.. pretty much whatever you want. We can have a downstream plugin that does $stuff for us as a short-term workaround. |
@dtantsur sounds interesting - can you link any docs/examples? Could we modify /boot (sed the grub.cfg and/or drop some file in there) after IPA writes the OS image? What would the interface from ironic look like? |
It's a brand new feature, I'm going to work on examples this/next week.
Yes! But do you really need it to happen afterwards or do you need to modify
There are options. If we create an openshift-specific plugin, it may have no interface at all, just silently do what we need. If you could remind me what exactly we need to do (I'm a bit lost in the history of this issue), I can say specifically. |
Note FCOS and RHCOS don't use All these are things that |
I'm not sure it's going to happen, at least not in all scenarios. |
@jlebon can you link any docs on how this works in an automated environment? E.g if you pxe boot the installer ramdisk, how does this data get provided to coreos-installer for a non-interactive deploy? As @dtantsur says there is some uncertainty/challenge in making this change (hence I mention it as a long term plan above), so we'll need to investigate further. We'll need to document a workaround in the meantime - does anything exist (upstream or downstream) that describes the process of using virt-customize to drop additional network configuration in /boot? (I know the virt-customize commands, but not certain on the network file format expected, and we need something downstream to share with $customer). |
We don't have good docs yet for that right now, but in an automated environment, you would script a systemd unit which calls If you're rolling your own code, you can of course do the same thing |
|
Since coreos/fedora-coreos-config#426 merged (fixes coreos/fedora-coreos-tracker#443) it feels like the question of how to "Support disabling initramfs networking via Ignition" is answered.
The caveat to this is that there are some platforms that only get their Ignition configuration by using the network, so it's not really possible to disable it on those platforms, nor would you want to. Shall we close this issue out? |
I pushed openshift/enhancements#467 so we can discuss some ideas around the "flattened ignition" solution, which (where appropriate, and in particular for IPI baremetal) could enable a MCO managed Ignition config that doesn't have any remote references
It seems the the conclusion of the discussion so far is that we won't fix this by adding some new ignition interface for early network config, so that seems reasonable to me. |
|
Thanks @hardys - Will follow along the discussion in openshift/enhancements#467 |
|
For people following Fedora CoreOS: the work for coreos/fedora-coreos-tracker#443 was first introduced in testing stream release |
Feature Request
Support running part of ignition before networking comes up
Environment
RHCOS
What hardware/cloud provider/hypervisor is being used to run Ignition?
Baremetal
Desired Feature
For baremetal IPI platform in OpenShift, we load ignition from disk. We would like to include network configuration that happens before NetworkManager starts. For example, we want to disable dhcp on some interfaces. If we can't do that, then on some hosts that could have upwards of 10 (!) interfaces, booting takes an extraordinary long time.
Also, when an ignition contains an 'append' with a network source, it's conceivable we need to do some pre-configuration to make that network endpoint accessible (for example, we need to configure a VLAN on an interface).
Other Information
It's almost like we need a pre-ignition tool (preflight?) analogous to what afterburn is doing post-ignition.
The text was updated successfully, but these errors were encountered: