[RFE/Fix]: Smooth out SELinux rough edges in Flatcar

[JAORMX]

Current situation

Given issues with having the whole system labeled in the past, not every file in the base distribution is labeled with the appropriate SELinux type. On the other hand, crucial boot-time tooling such as torcx has historically had issues with SELinux enforcing at boot time.

In order to provide a smoother experience in a SELinux enforcing deployment, it would be ideal to get these issues sorted out.

As of file system labeling, an example is the lack of labeling for sssd-related files. Another example is the lack of labeling of everything under /usr/bin. Labeling these would allow us to have more thorough policies and to start enforcing appropriate domain transitions from these binaries.

We can see that the default context is correct while the actual type is unlabeled.

$ ls -Z /usr/bin/cat
system_u:object_r:unlabeled_t:s0 /usr/bin/cat
$ selabel_lookup -v -k /usr/bin/cat
Default context: system_u:object_r:bin_t:s0
$  ls -Za /var/log/sssd/
system_u:object_r:unlabeled_t:s0 .    system_u:object_r:var_log_t:s0 ..

Goals

• Enable Flatcar to boot on a SELinux enforcing mode without hiccups
• Appropriately label files accordingly to allow enforcing policies more thoroughly

Non-Goals

• Set SELinux to enforcing by default: While this would be a better security default, this change would be too disruptive to bring in one go.

Impact

This would require us to revisit the SELinux packages that are provided and fix the necessary bits for the policy to work in Flatcar.

This might also require some updates to more recent versions of refpolicy for us to get the latest changes in policy and support containers better.

Ideal future situation

• One is able to turn SELinux to enforcing at boot time and be able to run relevant workloads with the required confinement.
• One is able to limit workloads to access parts of the system in a safe manner. This would require us to label the whole filesystem accordingly.

Implementation options

• add setfiles to the initramfs and enable the selinuxRelabel Ignition feature to relabel files created by Ignition (dracut/ignition: add setfiles to initramfs bootengine#42)

Additional information

Related issues:

• Kube Flannel (> 0.14.1) does not start in SELinux enforcing mode #635
• Can't install Kubernetes >=1.22 with RKE due to missing SELinux custom policies #598
• All Docker containers fail to start after update to 2983.2.0 #544
• SELinux prevents usage of execsnoop in enforcing mode #509
• [RFE] new package: sec-policy/selinux-container #479
• build_library: set correct SELinux contexts in final images scripts#368
• [RFE] rework SELinux patches #695
• Flannel >= 0.17.0 is crashing with enforced SELinux #779
• when selinux is enforcing unsigned kernel modules can't be loaded #783
• Improve SELinux docs #303
• [RFE] Enforce SELinux on all Mantle tests #1186

[JAORMX]

also, getting a newer reference of refpolicy would help, as we really need spc_t to be available.

[tormath1]

I started to investigate on relabeling the whole system during the build of the image.

[perfinion]

I started to investigate on relabeling the whole system during the build of the image.

Some quick thoughts that might be useful here:

1. setfiles has a -r /alt_root/ (instead of chroot'ing).
2. you probably want to use setfiles -D Set or update any directory SHA1 digests. Use this option to enable usage of the security.sehash extended attribute. it is originally from android. it puts the hash of the policy in the xattr so if there is no policy change it quits early. Android uses this to just relabel the entire fs every boot since there is no time penalty anymore.

so basically you do setfiles -Dv -r /mnt/gentoo /mnt/gentoo/etc/selinux/mcs/contexts/files/file_contexts /mnt/gentoo when building the image, then for handling updates the initrd or early during boot you can do restorecon -rD /
See NOTES at the bottom of man 8 restorecon for more info on sehash.

[alban]

Thanks for working on this.

For the record, I had an issue with the Security Profiles Operator Daemon and Flatcar 3185.0.0. I will post below in case it helps someone with the same problem (https://xkcd.com/979/)

It failed to start with:

   message: 'failed to create containerd task: failed to create shim task: OCI
     runtime create failed: runc create failed: unable to start container process:
     error during container init: write /proc/self/attr/keycreate: invalid argument:
     unknown'
   reason: StartError

The current workaround:

• kubectl edit SecurityProfilesOperatorDaemon -n security-profiles-operator spod
• selinuxTypeTag: unconfined_t

[JAORMX]

For the record, the SPO (Security Profiles Operator) runs the daemon with spc_t by default. This doesn't work with the current refpolicy version that ships with Flatcar. An update is needed.

[bitfisher]

Any progress so far?

[tormath1]

Hi @bitfisher,

No real progress on this task, we got some contributions and selinux-container is now available upstream (#479 (comment)) so we could try to integrate it among the refpolicy upgrade.

[bitfisher]

Any updates?

[tormath1]

Hi @bitfisher,

We started to work on this recently - see the office hours dedicated section: https://youtu.be/r3VPy5tsKmE?t=553 and the associated PRs: flatcar-archive/coreos-overlay#1993 (comment)

[bitfisher]

Hi @tormath1,

Glad to hear there is progess!

Can you give any estimate on when this will be available?

[alirezaghey]

Note to internet strangers, that may bump on this thread: If you try to install k8s with Cilium CNi on Flatcar, it won't work with the current implementation of SELinux, even in permissive mode. Disable SELinux. Cheers!

[tormath1]

@bitfisher a big piece is about to be done: relabeling the whole system. Once done, we must chase down a few denials raised by the CI and finally run all the tests in the CI with SELinux enforced at boot to be sure that everything works correctly.
This is quite complicated to give an estimate as it's not a regular package update. As usual, I keep folks updated on the Matrix channel and I'll comment from now into this issue for reached milestones.

[bitfisher]

Sorry for the late response. I've missed your comment :(

Glad to hear, that there is some progress.

We are still stuck to k8s < 1.22 here :(

BTW ... Is SELinux used if k8s uses containerd directly and k8s control plane runs via docker with SELinux diabled?
How could i check this? Sorry, i'm a real SELinux noob :(