Include systemd-logind for inhibit functionality #3308
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bcressey
reviewed
Jul 31, 2023
There was a problem hiding this comment.
Can you diff the features and options output for this? The build log will dump out a config like this:
#20 14.08 systemd 250
#20 14.08·
#20 14.08 build mode : release
#20 14.08 split /usr : False
#20 14.08 split bin-sbin : True
...
I'd also like to see a comparison of the package contents (rpm -qlp build/rpms/...) before and after to understand all the new units that are coming in.
|
Still collecting data, but here is the RPM difference after this change: |
|
Diff of features and options section: $ diff trimmed-systemd-current.output trimmed-systemd-updated.output
1d0
<
64,65c63,64
< enabled : ACL, SECCOMP, SELinux, blkid, libfdisk, efi, networkd, pstore, randomseed, repart, systemd-analyze, sysusers, tmpfiles, kmod, ldconfig, gshadow, link-udev-shared, link-systemctl-shared
< disabled : AUDIT, AppArmor, IMA, PAM, SMACK, elfutils, gcrypt, gnutls, libbpf, libcryptsetup, libcryptsetup-plugins, libcurl, libfido2, libidn, libidn2, libiptc, microhttpd, openssl, p11kit, pcre2, pwquality, qrencode, tpm2, xkbcommon, zstd, lz4, xz, zlib, bzip2, backlight, binfmt, bpf-framework, coredump, environment.d, gnu-efi, firstboot, hibernate, homed, hostnamed, hwdb, importd, initrd, kernel-install, localed, logind, machined, nss-myhostname, nss-mymachines, nss-resolve, nss-systemd, oomd, portabled, quotacheck, resolve, rfkill, sysext, timedated, timesyncd, userdb, vconsole, xdg-autostart, idn, polkit, nscd, legacy-pkla, dbus, glib, tpm, man pages, html pages, man page indices, SysV compat, compat-mutable-uid-boundaries, utmp, adm group, wheel group, debug hashmap, debug mmap cache, debug siphash, valgrind, trace logging, install tests, link-networkd-shared, link-timesyncd-shared, link-boot-shared, fexecve, standalone-binaries, static-libsystemd, static-libudev, cryptolib, DNS-over-TLS
---
> enabled : ACL, SECCOMP, SELinux, blkid, libfdisk, efi, logind, networkd, pstore, randomseed, repart, systemd-analyze, sysusers, tmpfiles, kmod, ldconfig, gshadow, link-udev-shared, link-systemctl-shared
> disabled : AUDIT, AppArmor, IMA, PAM, SMACK, elfutils, gcrypt, gnutls, libbpf, libcryptsetup, libcryptsetup-plugins, libcurl, libfido2, libidn, libidn2, libiptc, microhttpd, openssl, p11kit, pcre2, pwquality, qrencode, tpm2, xkbcommon, zstd, lz4, xz, zlib, bzip2, backlight, binfmt, bpf-framework, coredump, environment.d, gnu-efi, firstboot, hibernate, homed, hostnamed, hwdb, importd, initrd, kernel-install, localed, machined, nss-myhostname, nss-mymachines, nss-resolve, nss-systemd, oomd, portabled, quotacheck, resolve, rfkill, sysext, timedated, timesyncd, userdb, vconsole, xdg-autostart, idn, polkit, nscd, legacy-pkla, dbus, glib, tpm, man pages, html pages, man page indices, SysV compat, compat-mutable-uid-boundaries, utmp, adm group, wheel group, debug hashmap, debug mmap cache, debug siphash, valgrind, trace logging, install tests, link-networkd-shared, link-timesyncd-shared, link-boot-shared, fexecve, standalone-binaries, static-libsystemd, static-libudev, cryptolib, DNS-over-TLS
142c141
< logind : false
---
> logind : true |
stmcginnis
force-pushed
the
systemd-inhibit
branch
from
61a8afd
to
a13707c
Compare
|
Updated to address comments. After some trial and error and testing, new rpm file diff: Same testing as mentioned in original description. Just for more data, here's what the systemd-logind process looks like: |
stmcginnis
force-pushed
the
systemd-inhibit
branch
from
a13707c
to
5ac573b
Compare
|
Rebased on the latest Had previously tested with the combined changes, but did another validation test with newest build: bash-5.1# systemd-inhibit --list
No inhibitors.
bash-5.1# apiclient apply <<EOF
> [settings.kubernetes]
> shutdown-grace-period = "15m"
> shutdown-grace-period-for-critical-pods = "10m"
> EOF
bash-5.1# systemd-inhibit --list
WHO UID USER PID COMM WHAT WHY MODE
kubelet 0 root 3954 kubelet shutdown Kubelet needs time to handle node shutdown delay
1 inhibitors listed.
bash-5.1# journalctl -u kubelet | grep shutdown
Aug 03 20:54:51 ip-192-168-63-177.us-east-2.compute.internal kubelet[3954]: I0803 20:54:51.726760 3954 nodeshutdown_manager_linux.go:137] "Creating node shutdown manager" shutdownGracePeriodRequested="15m0s" shutdownGracePeriodCriticalPods="10m0s" shutdownGracePeriodByPodPriority=[{Priority:0 ShutdownGracePeriodSeconds:300} {Priority:2000000000 ShutdownGracePeriodSeconds:600}] |
|
Do you mind sharing another diff of the rpm after the rebase? ( |
|
Here is the diff with --- systemd-base.txt 2023-08-04 14:09:34.576651598 +0000
+++ systemd-inhibit.txt 2023-08-04 14:09:45.592607534 +0000
@@ -11,6 +11,7 @@
/x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/systemd-dissect
/x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/systemd-escape
/x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/systemd-id128
+/x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/systemd-inhibit
/x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/systemd-machine-id-setup
/x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/systemd-mount
/x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/systemd-notify
@@ -75,6 +76,7 @@
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/bluetooth.target
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/boot-complete.target
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/ctrl-alt-del.target
+/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/dbus-org.freedesktop.login1.service
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/dev-hugepages.mount
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/dev-mqueue.mount
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/exit.target
@@ -96,6 +98,7 @@
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/modprobe@.service
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/multi-user.target.wants
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/multi-user.target.wants/getty.target
+/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/multi-user.target.wants/systemd-logind.service
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/network-online.target
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/network-pre.target
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/network.target
@@ -173,6 +176,7 @@
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/systemd-journald@.service
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/systemd-journald@.socket
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/systemd-kexec.service
+/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/systemd-logind.service
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/systemd-machine-id-commit.service
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/systemd-modules-load.service
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/systemd-network-generator.service
@@ -220,6 +224,7 @@
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/systemd-fsck
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/systemd-growfs
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/systemd-journald
+/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/systemd-logind
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/systemd-makefs
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/systemd-modules-load
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/systemd-network-generator
@@ -302,6 +307,7 @@
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/udev/rules.d/70-joystick.rules
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/udev/rules.d/70-memory.rules
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/udev/rules.d/70-mouse.rules
+/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/udev/rules.d/70-power-switch.rules
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/udev/rules.d/70-touchpad.rules
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/udev/rules.d/75-net-description.rules
/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/udev/rules.d/75-probe_mtd.rules
@@ -323,6 +329,7 @@
/x86_64-bottlerocket-linux-gnu/sys-root/usr/share/dbus-1/system-services
/x86_64-bottlerocket-linux-gnu/sys-root/usr/share/dbus-1/system-services/org.freedesktop.systemd1.service
/x86_64-bottlerocket-linux-gnu/sys-root/usr/share/dbus-1/system.d
+/x86_64-bottlerocket-linux-gnu/sys-root/usr/share/dbus-1/system.d/org.freedesktop.login1.conf
/x86_64-bottlerocket-linux-gnu/sys-root/usr/share/dbus-1/system.d/org.freedesktop.systemd1.conf
/x86_64-bottlerocket-linux-gnu/sys-root/usr/share/factory
/x86_64-bottlerocket-linux-gnu/sys-root/usr/share/factory/etc/issue |
bcressey
reviewed
Aug 4, 2023
Settings were added for kubelet to be configured with pod shutdown grace policy settings. This feature allows kubelet to hold off a system shutdown until it has had a chance to terminate and migrate pod workloads cleanly. Unfortunately, it was missed when these settings were added that the mechanism kubelet uses internally for this is to use systemd-inhibit that is part of systemd-logind. Without the systemd-logind functionality included in Bottlerocket, the attempt to start processes with inhibit protection would fail and shutdown calls would be immediate. This adds systemd-logind to the systemd build process to enable this functionality, excluding things like PAM and its CLI since those are not needed for our inhibit requirements. Signed-off-by: Sean McGinnis <stmcg@amazon.com>
stmcginnis
force-pushed
the
systemd-inhibit
branch
from
5ac573b
to
0db6c67
Compare
bcressey
approved these changes
Aug 4, 2023
arnaldo2792
approved these changes
Aug 4, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue number:
Closes #3305
Description of changes:
Settings were added for kubelet to be configured with pod shutdown grace policy settings. This feature allows kubelet to hold off a system shutdown until it has had a chance to terminate and migrate pod workloads cleanly.
Unfortunately, it was missed when these settings were added that the mechanism kubelet uses internally for this is to use systemd-inhibit that is part of systemd-logind. Without the systemd-logind functionality included in Bottlerocket, the attempt to start processes with inhibit protection would fail and shutdown calls would be immediate.
This adds systemd-logind to the systemd build process to enable this functionality, excluding things like PAM and its CLI since those are not needed for our inhibit requirements.
Testing done:
Checked initial inhibit list:
Configured Kubernetes graceful shutdown periods:
Checked
kubeletservice status and checked logs to verify the node shutdown manager picked up settings without any errors:Listed inhibited processes again and verified kubelet shows up:
bash-5.1# systemd-inhibit --list WHO UID USER PID COMM WHAT WHY MODE kubelet 0 root 4356 kubelet shutdown Kubelet needs time to handle node shutdown delay 1 inhibitors listed.Rebooted one of the nodes and made sure there were no errors reported.
Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.