persist state for wicked, chrony, and systemd #970

bcressey · 2020-07-07T20:12:27Z

Issue number:
#248 , #249 , #250

Description of changes:
Confine chrony, wicked, and dbus-broker via SELinux policy; move DHCP leases and clock measurements back to persistent storage. Also persist the journal and the random seed.

Testing done:
No AVC denials logged in dmesg or in the journal.

Processes are labeled as expected.

system_u:system_r:bus_t:s0         3023 ?        Ss     0:00 /usr/bin/dbus-broker-launch --scope system
system_u:system_r:bus_t:s0         3067 ?        S      0:00 dbus-broker --log 4 --controller 9 --machine-id 990978842db9412194d093dc75af968b --max-
system_u:system_r:network_t:s0     3124 ?        Ss     0:00 /x86_64-bottlerocket-linux-gnu/sys-root/usr/libexec/wicked/bin/wickedd-auto4 --systemd
system_u:system_r:network_t:s0     3136 ?        Ss     0:00 /x86_64-bottlerocket-linux-gnu/sys-root/usr/libexec/wicked/bin/wickedd-dhcp4 --systemd
system_u:system_r:network_t:s0     3155 ?        Ss     0:00 /x86_64-bottlerocket-linux-gnu/sys-root/usr/libexec/wicked/bin/wickedd-dhcp6 --systemd
system_u:system_r:network_t:s0     3218 ?        Ss     0:00 /x86_64-bottlerocket-linux-gnu/sys-root/usr/sbin/wickedd --systemd --foreground
system_u:system_r:network_t:s0     3280 ?        Ss     0:00 /x86_64-bottlerocket-linux-gnu/sys-root/usr/sbin/wickedd-nanny --systemd --foreground
system_u:system_r:clock_t:s0       3555 ?        Ss     0:00 /usr/sbin/chronyd -d -F -1
system_u:system_r:clock_t:s0       3610 ?        S      0:00 /usr/sbin/chronyd -d -F -1

Files are labeled as expected.

# ls -latrZ /var/lib/wicked
-rwx------. 1 root root system_u:object_r:lease_t:s0   38 Jul  7 17:39 iaid.xml
-rwx------. 1 root root system_u:object_r:lease_t:s0   55 Jul  7 17:39 duid.xml
-rw-------. 1 root root system_u:object_r:lease_t:s0  996 Jul  7 20:03 lease-eth0-dhcp-ipv4.xml

# ls -latrZ /var/lib/chrony/
-rw-r--r--. 1 chrony chrony system_u:object_r:measure_t:s0 1121 Jul  7 20:09 169.254.169.123.dat
-rwx------. 1 chrony chrony system_u:object_r:measure_t:s0   42 Jul  7 20:09 drift

# ls -latrZ /var/lib/systemd/
-rwxr-xr-x.  1 root root system_u:object_r:state_t:s0  512 Jul  8 14:51 random-seed
drwxr-xr-x.  2 root root system_u:object_r:state_t:s0 4096 Jul  8 14:51 coredump
drwxr-xr-x.  2 root root system_u:object_r:state_t:s0 4096 Jul  8 14:51 catalog

# ls -latrZ /var/log/journal/ec274cd5add60d0d045b205a4a4317a8/
-rw-r-----. 1 root systemd-journal system_u:object_r:state_t:s0 8388608 Jul  8 14:58 system.journal

The machine ID is consistent across reboots and stop/start for both m4.xlarge and m5.xlarge.

/* m4.xlarge */
# cat /sys/hypervisor/uuid
ec274cd5-add6-0d0d-045b-205a4a4317a8

# cat /etc/machine-id
ec274cd5add60d0d045b205a4a4317a8

/* m5.xlarge */
# cat /etc/machine-id
ec2e505a672c97f59ba650eb81e9ed6d

# cat /sys/class/dmi/id/product_uuid
ec2e505a-672c-97f5-9ba6-50eb81e9ed6d

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

Before saving clock measurements and DHCP leases to durable storage, we want to make sure they can't be modified by an untrusted workload. chronyd and wicked are both network-facing daemons, so we have the additional goal of limiting their access to the host OS, to mitigate the impact of a compromise. dbus-broker does not need to persist any state, but could be exposed to untrusted local input, and benefits from similar confinement. systemd also persists some state, such as the journal and the random seed, so we label that data as well to prevent unexpected changes. Signed-off-by: Ben Cressey <bcressey@amazon.com>

Now that clock measurements are protected by the SELinux policy, we can save them on durable storage so they are available after an OS update. Signed-off-by: Ben Cressey <bcressey@amazon.com>

Now that lease files are protected by the SELinux policy, we can save them on durable storage so they are available after an OS update. Signed-off-by: Ben Cressey <bcressey@amazon.com>

This partially reverts 58b0289. We do not use the systemd feature for private directories, so this change only affects systemd's own programs. Signed-off-by: Ben Cressey <bcressey@amazon.com>

We don't have a good way to persist the machine ID in a place that's accessible during early boot, so our best bet is to always generate the same ID. Both Xen and KVM can provide a UUID to identify guests, and systemd already uses the KVM value if available. This patch extends the logic to check for the Xen value as well. Signed-off-by: Ben Cressey <bcressey@amazon.com>

Signed-off-by: Ben Cressey <bcressey@amazon.com>

packages/systemd/9001-move-stateful-paths-to-ephemeral-storage.patch

packages/systemd/var-run-tmpfiles.conf

tjkirch

I was left confused after Jake's comment - we're not losing /var/cache on reboot, right? We rely on it for migrations, at least, and surely other things.

The testing focuses on SELinux, and does show the machine ID, but you confirmed that all these things are persisting over reboot? Old journal entries, same machine ID, chrony timings, wicked lease...

packages/systemd/9004-machine-id-setup-generate-stable-ID-under-Xen.patch

bcressey · 2020-07-08T18:08:10Z

I was left confused after Jake's comment - we're not losing /var/cache on reboot, right? We rely on it for migrations, at least, and surely other things.

Correct, /var/cache is not going away.

The testing focuses on SELinux, and does show the machine ID, but you confirmed that all these things are persisting over reboot? Old journal entries, same machine ID, chrony timings, wicked lease...

Yes, I've confirmed that the files are persisted across reboot.

bcressey · 2020-07-08T20:35:05Z

I've also tested a downgrade from a system with the stateful files into an older version. wicked and chrony were both stateless before and didn't expect files in /var/lib, so those were fine.

My concern was with journald, which previously defaulted to Storage=auto, which uses a persistent journal if /var/log/journal exists, and otherwise uses /run/log/journal. Since the persistent journal is created by the upgrade, it will still be there on the downgrade, and labeled with an SELinux label that didn't exist in the older policy.

It ended up working fine, though, since the policy allows for writes to files with no label or an unknown label.

jamieand

LGTM

This was linked to issues Jul 7, 2020

DHCP leases should be persisted on local storage #248

Closed

chrony drift files should be persisted on local storage #249

Closed

bcressey requested review from tjkirch and jamieand July 7, 2020 20:13

tjkirch approved these changes Jul 7, 2020

View reviewed changes

bcressey added 6 commits July 8, 2020 14:22

chrony: move clock state back to persistent storage

99f13bb

Now that clock measurements are protected by the SELinux policy, we can save them on durable storage so they are available after an OS update. Signed-off-by: Ben Cressey <bcressey@amazon.com>

wicked: move interface state back to persistent storage

b730f19

Now that lease files are protected by the SELinux policy, we can save them on durable storage so they are available after an OS update. Signed-off-by: Ben Cressey <bcressey@amazon.com>

systemd: move stateful paths back to persistent storage

10bb585

This partially reverts 58b0289. We do not use the systemd feature for private directories, so this change only affects systemd's own programs. Signed-off-by: Ben Cressey <bcressey@amazon.com>

systemd: configure journal for persistent storage

4d6679a

Signed-off-by: Ben Cressey <bcressey@amazon.com>

bcressey force-pushed the persist-state branch from 5545eb1 to 4d6679a Compare July 8, 2020 14:25

bcressey changed the title ~~persist state for wicked and chrony~~ persist state for wicked, chrony, and systemd Jul 8, 2020

bcressey linked an issue Jul 8, 2020 that may be closed by this pull request

systemd journal and machine ID should be persisted on local storage #250

Closed

2 tasks

bcressey requested a review from tjkirch July 8, 2020 16:17

jahkeup reviewed Jul 8, 2020

View reviewed changes

packages/systemd/9001-move-stateful-paths-to-ephemeral-storage.patch Show resolved Hide resolved

packages/systemd/var-run-tmpfiles.conf Show resolved Hide resolved

tjkirch reviewed Jul 8, 2020

View reviewed changes

packages/systemd/9004-machine-id-setup-generate-stable-ID-under-Xen.patch Show resolved Hide resolved

tjkirch approved these changes Jul 8, 2020

View reviewed changes

jamieand approved these changes Jul 8, 2020

View reviewed changes

bcressey merged commit 556ae73 into bottlerocket-os:develop Jul 9, 2020

bcressey deleted the persist-state branch July 9, 2020 21:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

persist state for wicked, chrony, and systemd #970

persist state for wicked, chrony, and systemd #970

bcressey commented Jul 7, 2020 •

edited

Loading

tjkirch left a comment

bcressey commented Jul 8, 2020

bcressey commented Jul 8, 2020

jamieand left a comment

persist state for wicked, chrony, and systemd #970

persist state for wicked, chrony, and systemd #970

Conversation

bcressey commented Jul 7, 2020 • edited Loading

tjkirch left a comment

Choose a reason for hiding this comment

bcressey commented Jul 8, 2020

bcressey commented Jul 8, 2020

jamieand left a comment

Choose a reason for hiding this comment

bcressey commented Jul 7, 2020 •

edited

Loading