migrate akmod keys from repo to org level

[bsherman]

We need to migrate AKMOD_PRIVKEY and AKMOD_PUBKEY to be org-level secrets rather than nvidia repo level.

Reason: we plan to use the same single key for akmods signing of both our current nvidia akmod and our future akmods.

[castrojo]

After talking to josh it ends up we never intended for a multi-repo use case and pretty much comes right out of the prototype. We missed checking this as part of moving to beta, we should have caught this.

We don't have the key, we need to generate a new one. However, Josh says we can have existing users run a command to migrate to the new key.

Then I'll take an item to sort out key escrow or whatever we're supposed to be doing. Then we can ship a fix in a justfile or something to find a least-worst migration solution. And then probably generate a new key and rip the bandaid off for F39?

The reason to do this (and I think we should) is we need to have akmods in it's own repo so that we can split that off as a generic solution that we need to get other out of tree kernel modules that we want.

[bsherman]

Thank you VERY much for tracking that down with Josh. Good that we at least know what the situation is.

I can continue the generic akmods work, but once its ready, that's pretty much the time to rip off the bandaid, not F39 release. I'm not really sure what you are thinking for formal key escrow, but I'm guessing that could take longer than the remaining akmods dev effort at this point. :-)

[castrojo]

Also on further thought, I believe our line of thinking was "why would we need an org secret for this keep that contained in that one repo, reduce the scope."

[bsherman]

Also on further thought, I believe our line of thinking was "why would we need an org secret for this keep that contained in that one repo, reduce the scope."

No worries. Clearly you just needed a contributor with bigger vision. 😂

[bsherman]

Joking aside... here's the serious thoughts about how to proceed:

(:heavy_check_mark: strikeout indicates completion)

1. ✔️ create new keypair ASAP and get them into org-level secrets (I'm unsure if this is blocked by escrow or if @castrojo can do that async)
2. ✔️ add new public signing key to *-main images with the new filename so it's available for users to start importing
3. create a script/just command which is also in *-main for easy use
4. create Github, Discord and Website announcement about this topic... declaring a date for nvidia signing key cutover... pointing to the instructions (on website) how to import new key... explaining consequences for SecureBoot nvidia users if they don't
5. [maybe] get annoying, and create a zenity script (or something) that we attempt to force run for all users who use SecureBoot and nvidia, pointing them to the website announcement/instructions

That last step needs a lot of thought with regard to usefulness, viability, and how to make sure it isn't ANOTHER source of problems.

[castrojo]

Yeah or maybe we can do something like ship an enabled systemd service that checks to see if you're on the old key every X hours and if you are xdg-open a link to the migration instructions, or /etc/motd, or whatever we think would catch it.

[Arcitec]

People who use uBlue know it's an alpha. We say it everywhere. Waiting for F39 would be a waste of our time. Holding back progress, new akmods and infrastructure for an arbitrary release date lol.

I am definitely in favor of ripping the bandaid soon, and the path suggested by @bsherman sounds great. Just needs that migration path for existing users.

[bsherman]

create new keypair ASAP and get them into org-level secrets (I'm unsure if this is blocked by escrow or if @castrojo can do that async)

Please create new key in org-level secrets with name AKMOD_PRIVKEY_20230517 and place the pub key in ublue-os/main repo at certs/public_key.der.

[bsherman]

@castrojo completed step 2 (and fulfilled my previous comment request) with ublue-os/main#212

[Arcitec]

Good thing is... Nothing is really broken forever. Even if the update rolls out to someone who hasn't migrated, they can use the following process to recover their system:

1. Disable Secure Boot in BIOS (since that's what prevents NVIDIA from loading when the signing key isn't trusted).
2. Boot back inside the OS, where you'll now have a properly loading nvidia driver.
3. Remove the nvidia kernel kargs to allow nouveau to be loaded at boot again.
4. Re-enable secure boot (to use nouveau + secure boot instead of the NVIDIA driver).
5. Enroll the new key.
6. Finally enable kargs again, to re-enable NVIDIA drivers.

So it is more an issue of getting the word out. We want to avoid the black screens in the first place, by making sure people have the new key already.

[castrojo]

I am in and out today so if someone who has time on an nvidia image could doublecheck that the cert is on there now that'd be great.

[bsherman]

I am in and out today so if someone who has time on an nvidia image could doublecheck that the cert is on there now that'd be great.

I'm sorry, I didn't communicate very well and I was simultaneously trying to handle some issues at work, so I overlooked something. We did add the new cert into the repo, but it's not yet built into the image. I'll PR that quickly.

[bsherman]

My PR to correctly implement item 2 from #100 (comment)

#102

I've pretty much just committed to June 17 as signing cutover day.

[Arcitec]

@bsherman

I've pretty much just committed to June 17 as signing cutover day.

100% agreed, great decision.

Anyone have any ideas how we're going to notify people? There's always that process to repair all affected systems, but if there's some non-intrusive way we can figure out for notifying users, that would be better.

For example: In startingpoint, we use the following layout to install a small profile that drops an autostart program into every user's folder:

https://github.com/ublue-os/startingpoint/blob/template/usr/share/ublue-os/firstboot/launcher/launcher-flowchart.png

That technique would also work to drop in a small script that runs "notify-send" to send a desktop notification about it. Or even a small GTK python app that pops up with the news.

It depends on how serious we think this news is, because installing an announcement app like that on systems will annoy some people more than a black screen and a "ublue.it website - check news - see the fix" would.

[castrojo]

Ok from chat, next steps:

1. Test signing with the new key on a new install (can someone with an nvidia machine to spare test this?)
2. Add the oneliner fix to the README (In progress), this should also ensure new installs are done with the new key.
3. Add the instructions from this comment in the install page/FAQ to catch anyone who misses it: migrate akmod keys from repo to org level #100 (comment)

[bsherman]

1. Test signing with the new key on a new install (can someone with an nvidia machine to spare test this?)

I've done a few tests:

✔️ I've verified that the new key is present on an image derived from ublue-os/silverblue-nvidia:38 - so that's good.

✔️ I've moved files around in my local git checkout to force a build of nvidia modules with new keys. I've confirmed that, when new keys are in proper location, nvidia drivers do get signed with the new key.

😢 One minor complaint... Though the new key has a nicer O = Universal Blue, OU = akmods and Jorge's email for a contact... It could be kinda nice if our CN was more descriptive too... Ideally CN = uBlue akmods signing CA .

Subject: O = Universal Blue, OU = akmods, emailAddress = <jorge's email>, L = None, ST = None, C = XX, CN = akmods local signing CA

Below you can see the output of modinfo for a kmod signed with new key... the sig_key matches our new key. As does the signer: field, but signer is so generic, we can't easily tell if a user has the new key or not, we'll have to rely on sig_key.

@castrojo I don't want you do do extra work, but, it's probably slightly better for long term trustworthiness and debugging if we correct that CN now, before we are stuck with it.

I leave it to you, though, as we aren't sunk without the fix, it's just nicer.

modinfo nvidia-drm.ko.xz
filename:       /lib/modules/6.2.15-300.fc38.x86_64/extra/nvidia/nvidia-drm.ko.xz
version:        530.41.03
supported:      external
license:        MIT
srcversion:     EC27D424D89D736970E49AA
alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
depends:        nvidia-modeset
retpoline:      Y
name:           nvidia_drm
vermagic:       6.2.15-300.fc38.x86_64 SMP preempt mod_unload
sig_id:         PKCS#7
signer:         akmods local signing CA
sig_key:        41:0C:6D:27:13:56:49:55:76:DF:E8:A1:74:A4:A4:22:2E:98:0A:91
sig_hashalgo:   sha256
--- snip out full signature for brevity ---
parm:           modeset:Enable atomic kernel modesetting (1 = enable (default), 0 = disable) (bool)

[castrojo]

What are the implications of the CN not being filled in?

[bsherman]

What are the implications of the CN not being filled in?

It's simply not as easy to read which key was used to sign a given kmod.

akmods local signing CA is the default string, so any user who generates their own key will see the same "signer" string on both their default key and on our "official" key. So it's confusing to know which was which. The sig_key does provide the answer, but it's a lot harder to read clearly and compare to know what's what.