Move to conditional networking #426

jlebon · 2020-05-25T18:48:23Z

We have all the piece in place now to move to conditional networking. So
let's drop the firstboot kargs, as well as
coreos-liveiso-network-kargs.service, which is no longer needed (i.e.
the live ISO will now enable initrd networking as required given the
embedded Ignition config).

Fixes: coreos/fedora-coreos-tracker#443

jlebon · 2020-05-25T18:48:46Z

Requires: coreos/coreos-assembler#1373
Requires: coreos/ignition#956
Requires: coreos/ignition-dracut#164

jlebon · 2020-05-26T13:36:02Z

Rebased this on top of #427!

darkmuggle

💯 I really like this.

jlebon · 2020-06-17T20:39:49Z

This will be the final ratcheting point for FCOS. Will add a test so we can validate it before merging.

dustymabe

LGTM

jlebon · 2020-06-18T19:12:21Z

Almost there! All remaining patches are merged, we just need an Ignition release at this point.

dustymabe · 2020-07-07T21:28:07Z

Almost there! All remaining patches are merged, we just need an Ignition release at this point.

Do we at least have any plans for that release to happen? /me really wants this in 😄

jlebon · 2020-07-10T13:42:55Z

Do we at least have any plans for that release to happen? /me really wants this in smile

I think once coreos/ignition#960 is in, we should cut a release. /cc @bgilbert @arithx

dustymabe · 2020-07-14T15:46:54Z

New ignition version just landed! #513

jlebon · 2020-07-14T20:16:14Z

Rebased this and marking ready for review! Still sanity-checking ISO behaviour locally as well.

dustymabe · 2020-07-14T20:25:06Z

I'll take a re-look at this PR as well and try to build locally

jlebon · 2020-07-14T21:08:30Z

Still sanity-checking ISO behaviour locally as well.

Working well on FCOS!

But yeah I think for now we'll need to keep the liveiso-network-kargs bits for RHCOS until we move to spec3 there and gain the Ignition fetch-offline work. Will just tweak those to make them active on RHCOS only.

dustymabe · 2020-07-14T22:36:43Z

This seems to be working well in my testing too! Though we did find #514 which needs to be addressed (though not in this PR). There is one subtle change here, which is if people were relying on networking configuration getting persisted into the real root and they were using ip= kargs for that in the past then they'll now need to make sure they also add rd.neednet=1, otherwise their ip= settings won't get applied persistently to the real root. Maybe a good point for a release note.

But yeah I think for now we'll need to keep the liveiso-network-kargs bits for RHCOS until we move to spec3 there and gain the Ignition fetch-offline work. Will just tweak those to make them active on RHCOS only.

Does that mean you're going to do another push to this PR?

cgwalters · 2020-07-14T23:48:51Z

There is one subtle change here, which is if people were relying on networking configuration getting persisted into the real root and they were using ip= kargs for that in the past then they'll now need to make sure they also add rd.neednet=1, otherwise their ip= settings won't get applied persistently to the real root. Maybe a good point for a release note.

Nice catch! Hmm...could we distinguish between "our default kargs" and "kargs provided by user" in that case though?

dustymabe · 2020-07-15T03:07:43Z

There is one subtle change here, which is if people were relying on networking configuration getting persisted into the real root and they were using ip= kargs for that in the past then they'll now need to make sure they also add rd.neednet=1, otherwise their ip= settings won't get applied persistently to the real root. Maybe a good point for a release note.

Nice catch! Hmm...could we distinguish between "our default kargs" and "kargs provided by user" in that case though?

We might be able to do some trickery (@jlebon would probably know best since he did this most recent round of work), but I'm on the fence. Since it's a straightforward change I think I'd be more inclined to request the users who need that to just add the rd.neednet=1 or even better use it as an opportunity to encourage them to put the config in the ignition config instead since they obviously don't need networking in the initramfs.

dustymabe · 2020-07-15T14:21:45Z

the fix for #514 seems to be working great!

We have all the piece in place now to move to conditional networking. So let's drop the `rd.neednet=1` firstboot karg. Also don't enable coreos-liveiso-network-kargs.service on FCOS since it's no longer needed (i.e. the live ISO will now enable initrd networking as required given the embedded Ignition config). On RHCOS, we still need it for now until we move to spec3. Then we can remove the service and script completely. Fixes: coreos/fedora-coreos-tracker#443

We shouldn't use `/run/ignition.json` to determine whether a user config was provided since it's implementation details. Instead, use the new official journal messages that Ignition emits. This is complicated by the fact that we need to support RHCOS, where the journal messages haven't been backported. Use the fact that we always have a base config to key off of whether to use the old behaviour vs the new one. (More accurately, we'd want to check for coreos/ignition#1002, but there's no easy way to do this from the outside. Alternatively we can check the Ignition version, though that's deeply nested under `/usr/lib/dracut/...`). Anyway, this should be temporary until RHCOS moves to spec v3. Closes: coreos#514

jlebon · 2020-07-15T18:45:04Z

OK, the fix for #514 ended up being more involved again because RHCOS has older Ignition. See commit message for full details.

Tested this on both FCOS and RHCOS both with and without an embedded Ignition config.

dustymabe · 2020-07-15T18:48:31Z

OK, the fix for #514 ended up being more involved again because RHCOS has older Ignition. See commit message for full details.

Tested this on both FCOS and RHCOS both with and without an embedded Ignition config.

👍

dustymabe · 2020-07-15T18:48:43Z

Still LGTM

bgilbert · 2020-07-15T18:56:52Z

The coreos.inst wrapper script appends rd.neednet=1 if it's forwarding any networking kargs to the first boot. AIUI that's still okay? We'll need to update the README though.

bgilbert

LGTM

jlebon · 2020-07-15T19:56:41Z

The coreos.inst wrapper script appends rd.neednet=1 if it's forwarding any networking kargs to the first boot. AIUI that's still okay?

Yup, that should be fine.

We've now reached the end of the conditional networking 🌈 rainbow 🌈. We can rip out the legacy default network firstboot kargs from cosa. These now live in the config repo. E.g.: https://github.com/coreos/fedora-coreos-config/blob/be456c437d4181435022ce47079b587f5bcb0319/overlay.d/05core/usr/lib/dracut/modules.d/15coreos-network/50-afterburn-network-kargs-default.conf For more details, see: coreos#1373 coreos/fedora-coreos-config#426

We've now reached the end of the conditional networking 🌈 rainbow 🌈. We can rip out the legacy default network firstboot kargs from cosa. These now live in the config repo. E.g.: https://github.com/coreos/fedora-coreos-config/blob/be456c437d4181435022ce47079b587f5bcb0319/overlay.d/05core/usr/lib/dracut/modules.d/15coreos-network/50-afterburn-network-kargs-default.conf For more details, see: #1373 coreos/fedora-coreos-config#426

We originally did this in coreos#326 because we wanted to support booting the live ISO without networking. This was solved on the initramfs side by the conditional networking work (coreos#426). But for the real root, this was still useful because if booting the ISO interactively on a system without any network, or a non-DHCP network, we didn't want the user to have to wait until the service timed out before getting a shell. The core issue however is that we're requesting `network-online.target` at all. It's an "active unit" which means that it's only pulled in the transaction, possibly delaying boot, if another systemd unit needs it. And ideally, no service would need it as per: https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ In our case, this unit was fedora-coreos-pinger. We drop that requirement here: coreos/fedora-coreos-pinger#41 With that, we no longer pull in `network-online.target` and so no longer delay reaching the console even if NetworkManager isn't able to get an active connection for whatever reason. This matches how it works on traditional Fedora as well. Having a short timeout actually also had a counterproductive effect in the automated install case. There, `coreos-installer.service` does pull in `network-online.target` (which with coreos/coreos-installer#565 we could consider dropping as advised by systemd, though we probably should bump the number of retries some more in that case), but because of the short timeout, we genuinely may not yet have the network fully up before we run (see https://bugzilla.redhat.com/show_bug.cgi?id=1967483).

We originally did this in #326 because we wanted to support booting the live ISO without networking. This was solved on the initramfs side by the conditional networking work (#426). But for the real root, this was still useful because if booting the ISO interactively on a system without any network, or a non-DHCP network, we didn't want the user to have to wait until the service timed out before getting a shell. The core issue however is that we're requesting `network-online.target` at all. It's an "active unit" which means that it's only pulled in the transaction, possibly delaying boot, if another systemd unit needs it. And ideally, no service would need it as per: https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ In our case, this unit was fedora-coreos-pinger. We drop that requirement here: coreos/fedora-coreos-pinger#41 With that, we no longer pull in `network-online.target` and so no longer delay reaching the console even if NetworkManager isn't able to get an active connection for whatever reason. This matches how it works on traditional Fedora as well. Having a short timeout actually also had a counterproductive effect in the automated install case. There, `coreos-installer.service` does pull in `network-online.target` (which with coreos/coreos-installer#565 we could consider dropping as advised by systemd, though we probably should bump the number of retries some more in that case), but because of the short timeout, we genuinely may not yet have the network fully up before we run (see https://bugzilla.redhat.com/show_bug.cgi?id=1967483).

We originally did this in coreos#326 because we wanted to support booting the live ISO without networking. This was solved on the initramfs side by the conditional networking work (coreos#426). But for the real root, this was still useful because if booting the ISO interactively on a system without any network, or a non-DHCP network, we didn't want the user to have to wait until the service timed out before getting a shell. The core issue however is that we're requesting `network-online.target` at all. It's an "active unit" which means that it's only pulled in the transaction, possibly delaying boot, if another systemd unit needs it. And ideally, no service would need it as per: https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ In our case, this unit was fedora-coreos-pinger. We drop that requirement here: coreos/fedora-coreos-pinger#41 With that, we no longer pull in `network-online.target` and so no longer delay reaching the console even if NetworkManager isn't able to get an active connection for whatever reason. This matches how it works on traditional Fedora as well. Having a short timeout actually also had a counterproductive effect in the automated install case. There, `coreos-installer.service` does pull in `network-online.target` (which with coreos/coreos-installer#565 we could consider dropping as advised by systemd, though we probably should bump the number of retries some more in that case), but because of the short timeout, we genuinely may not yet have the network fully up before we run (see https://bugzilla.redhat.com/show_bug.cgi?id=1967483). (cherry picked from commit dd54e8c)

We originally did this in #326 because we wanted to support booting the live ISO without networking. This was solved on the initramfs side by the conditional networking work (#426). But for the real root, this was still useful because if booting the ISO interactively on a system without any network, or a non-DHCP network, we didn't want the user to have to wait until the service timed out before getting a shell. The core issue however is that we're requesting `network-online.target` at all. It's an "active unit" which means that it's only pulled in the transaction, possibly delaying boot, if another systemd unit needs it. And ideally, no service would need it as per: https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ In our case, this unit was fedora-coreos-pinger. We drop that requirement here: coreos/fedora-coreos-pinger#41 With that, we no longer pull in `network-online.target` and so no longer delay reaching the console even if NetworkManager isn't able to get an active connection for whatever reason. This matches how it works on traditional Fedora as well. Having a short timeout actually also had a counterproductive effect in the automated install case. There, `coreos-installer.service` does pull in `network-online.target` (which with coreos/coreos-installer#565 we could consider dropping as advised by systemd, though we probably should bump the number of retries some more in that case), but because of the short timeout, we genuinely may not yet have the network fully up before we run (see https://bugzilla.redhat.com/show_bug.cgi?id=1967483). (cherry picked from commit dd54e8c)

We originally did this in coreos#326 because we wanted to support booting the live ISO without networking. This was solved on the initramfs side by the conditional networking work (coreos#426). But for the real root, this was still useful because if booting the ISO interactively on a system without any network, or a non-DHCP network, we didn't want the user to have to wait until the service timed out before getting a shell. The core issue however is that we're requesting `network-online.target` at all. It's an "active unit" which means that it's only pulled in the transaction, possibly delaying boot, if another systemd unit needs it. And ideally, no service would need it as per: https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ In our case, this unit was fedora-coreos-pinger. We drop that requirement here: coreos/fedora-coreos-pinger#41 With that, we no longer pull in `network-online.target` and so no longer delay reaching the console even if NetworkManager isn't able to get an active connection for whatever reason. This matches how it works on traditional Fedora as well. Having a short timeout actually also had a counterproductive effect in the automated install case. There, `coreos-installer.service` does pull in `network-online.target` (which with coreos/coreos-installer#565 we could consider dropping as advised by systemd, though we probably should bump the number of retries some more in that case), but because of the short timeout, we genuinely may not yet have the network fully up before we run (see https://bugzilla.redhat.com/show_bug.cgi?id=1967483).

) This expands the section describing permanent rollbacks, in order to make it more explicit how to deal with auto-updates and to show how to manually perform roll-forward steps if required. The scenario came up in a chat thread, and it is good to have the commands written for reference.

This was referenced May 25, 2020

Don't bring up networking in the initramfs on first boot by default coreos/fedora-coreos-tracker#443

Closed

grub: de-hardcode firstboot network configuration coreos/coreos-assembler#1373

Merged

jlebon force-pushed the pr/conditional-net branch from 4981677 to 849061f Compare May 26, 2020 13:35

jlebon force-pushed the pr/conditional-net branch from 849061f to d15bd41 Compare June 1, 2020 17:57

darkmuggle reviewed Jun 1, 2020

View reviewed changes

jlebon force-pushed the pr/conditional-net branch from d15bd41 to 9d0bd60 Compare June 9, 2020 21:27

jlebon force-pushed the pr/conditional-net branch from 9d0bd60 to 93ef4bd Compare June 17, 2020 20:37

jlebon mentioned this pull request Jun 17, 2020

Add new ignition-fetch-offline.service coreos/ignition-dracut#164

Merged

dustymabe approved these changes Jun 18, 2020

View reviewed changes

jlebon mentioned this pull request Jun 22, 2020

Support disabling initramfs networking via Ignition coreos/ignition#979

Closed

jlebon force-pushed the pr/conditional-net branch from 93ef4bd to a6fa2b4 Compare July 14, 2020 20:14

jlebon marked this pull request as ready for review July 14, 2020 20:15

jlebon mentioned this pull request Jul 15, 2020

interactive live ISO experience no longer works with Ignition 2.4.0 #514

Closed

jlebon force-pushed the pr/conditional-net branch from a6fa2b4 to 79111ec Compare July 15, 2020 13:12

jlebon marked this pull request as draft July 15, 2020 13:28

jlebon force-pushed the pr/conditional-net branch from 79111ec to 66dcf25 Compare July 15, 2020 13:31

jlebon added 2 commits July 15, 2020 14:43

jlebon marked this pull request as ready for review July 15, 2020 18:44

jlebon force-pushed the pr/conditional-net branch from 66dcf25 to 613f691 Compare July 15, 2020 18:44

bgilbert approved these changes Jul 15, 2020

View reviewed changes

jlebon merged commit 65de5e0 into coreos:testing-devel Jul 15, 2020

jlebon deleted the pr/conditional-net branch July 15, 2020 19:55

arithx mentioned this pull request Aug 6, 2020

networking and /boot/ignition/config.ign coreos/ignition#1052

Closed

jlebon mentioned this pull request Aug 21, 2020

Drop support for ignition-network-kcmdline coreos/coreos-assembler#1678

Merged

jlebon mentioned this pull request Jul 7, 2021

35coreos-live: stop overriding NetworkManager-wait-online timeout to 5s #1088

Merged

jlebon mentioned this pull request Jul 19, 2021

[rhcos-4.8] 35coreos-live: stop overriding NetworkManager-wait-online timeout to 5s #1120

Merged

jlebon mentioned this pull request Jul 19, 2021

[rhcos-4.7] 35coreos-live: stop overriding NetworkManager-wait-online timeout to 5s #1121

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move to conditional networking #426

Move to conditional networking #426

jlebon commented May 25, 2020

jlebon commented May 25, 2020

jlebon commented May 26, 2020

darkmuggle left a comment

jlebon commented Jun 17, 2020

dustymabe left a comment

jlebon commented Jun 18, 2020

dustymabe commented Jul 7, 2020

jlebon commented Jul 10, 2020

dustymabe commented Jul 14, 2020

jlebon commented Jul 14, 2020

dustymabe commented Jul 14, 2020

jlebon commented Jul 14, 2020

dustymabe commented Jul 14, 2020

cgwalters commented Jul 14, 2020

dustymabe commented Jul 15, 2020

dustymabe commented Jul 15, 2020

jlebon commented Jul 15, 2020

dustymabe commented Jul 15, 2020

dustymabe commented Jul 15, 2020

bgilbert commented Jul 15, 2020

bgilbert left a comment

jlebon commented Jul 15, 2020

Move to conditional networking #426

Move to conditional networking #426

Conversation

jlebon commented May 25, 2020

jlebon commented May 25, 2020

jlebon commented May 26, 2020

darkmuggle left a comment

Choose a reason for hiding this comment

jlebon commented Jun 17, 2020

dustymabe left a comment

Choose a reason for hiding this comment

jlebon commented Jun 18, 2020

dustymabe commented Jul 7, 2020

jlebon commented Jul 10, 2020

dustymabe commented Jul 14, 2020

jlebon commented Jul 14, 2020

dustymabe commented Jul 14, 2020

jlebon commented Jul 14, 2020

dustymabe commented Jul 14, 2020

cgwalters commented Jul 14, 2020

dustymabe commented Jul 15, 2020

dustymabe commented Jul 15, 2020

jlebon commented Jul 15, 2020

dustymabe commented Jul 15, 2020

dustymabe commented Jul 15, 2020

bgilbert commented Jul 15, 2020

bgilbert left a comment

Choose a reason for hiding this comment

jlebon commented Jul 15, 2020