[rawhide][branched] SELinux AVC denials causes the afterburn write to fail in cloud platforms

[aaradhak]

Describe the bug

In the recent rawhide & branched cloud platform builds, the kola tests are failing due to an error in the afterburn service.

harness.go:1823: mach.Start() failed: machine "i-0c58eeb79e70a7d44" failed basic checks: detected failed or stuck systemd units: some systemd units failed: afterburn-sshkeys@core.service; <nil>

On further debugging, it is found that the afterburn process attempted to write to the /var/home/core/.ssh/authorized_keys.d/ directory but was denied by SELinux . This denial caused the afterburn-sshkeys service to fail with a "Permission denied (os error 13)" error.

This seems to be like a selinux-policy issue.

Aug 20 17:00:43.703000 audit[1592]: AVC avc:  denied  { write } for  pid=1592 comm="afterburn" name="authorized_keys.d" dev="nvme0n1p4" ino=20971648 scontext=system_u:system_r:afterburn_t:s0 tcontext=unconfined_u:object_r:ssh_home_t:s0 tclass=dir permissive=0
Aug 20 17:00:43.705504 afterburn-sshkeys@core.service[1592]: Error: failed to run
Aug 20 17:00:43.705504 afterburn-sshkeys@core.service[1592]: Caused by:
Aug 20 17:00:43.705504 afterburn-sshkeys@core.service[1592]:     0: writing ssh keys
Aug 20 17:00:43.705504 afterburn-sshkeys@core.service[1592]:     1: failed to create temporary file
Aug 20 17:00:43.707051 afterburn-sshkeys@core.service[1592]:     2: Permission denied (os error 13) at path "/var/home/core/.ssh/authorized_keys.d/.afterburn-gRJMKD"
Aug 20 17:00:43.707269 init.scope[1]: afterburn-sshkeys@core.service: Main process exited, code=exited, status=1/FAILURE
Aug 20 17:00:43.707481 init.scope[1]: afterburn-sshkeys@core.service: Failed with result 'exit-code'.
Aug 20 17:00:43.707727 init.scope[1]: Failed to start afterburn-sshkeys@core.service - Afterburn (SSH Keys).

Apart from the above AVC denials, came across few other AVC denials in the journal log as below:

Aug 20 17:00:37.025779 kernel: audit: type=1400 audit(1724173234.341:4): avc:  denied  { getattr } for  pid=1367 comm="coreos-boot-mou" path="/run/coreos/bootfs_uuid" dev="tmpfs" ino=870 scontext=system_u:system_r:coreos_boot_mount_generator_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=1
Aug 20 17:00:37.025790 kernel: audit: type=1400 audit(1724173234.357:5): avc:  denied  { read } for  pid=1403 comm="cat" name="bootfs_uuid" dev="tmpfs" ino=870 scontext=system_u:system_r:coreos_boot_mount_generator_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=1
Aug 20 17:00:37.025801 kernel: audit: type=1400 audit(1724173234.357:6): avc:  denied  { open } for  pid=1403 comm="cat" path="/run/coreos/bootfs_uuid" dev="tmpfs" ino=870 scontext=system_u:system_r:coreos_boot_mount_generator_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=1
Aug 20 17:00:37.025813 systemd[1]: Populated /etc with preset unit settings.
Aug 20 17:00:37.029860 systemd[1]: initrd-switch-root.service: Deactivated successfully.

Reproduction steps

Start a pipeline job build of the kola cloud platforms.

• kola-aws
• kola-gcp
• kola-azure
• kola-openstack

Expected behavior

The SELinux policy to allow the afterburn process to write to the directory in question.

Actual behavior

kola tests fails with this error:

harness.go:1823: mach.Start() failed: machine "i-0c58eeb79e70a7d44" failed basic checks: detected failed or stuck systemd units: some systemd units failed: afterburn-sshkeys@core.service; <nil>

System details

Kola cloud platform pipeline jobs.
Streams - rawhide & branched

• kola-aws
• kola-gcp
• kola-azure
• kola-openstack

Butane or Ignition config

No response

Additional information

There's a similar afterburn issue that was filed against c9s that was fixed, but possibly the fixes there need to be brought to Fedora too: https://issues.redhat.com/browse/RHEL-49735

[aaradhak]

Opened a BZ for this selinux-policy avc denial - https://bugzilla.redhat.com/show_bug.cgi?id=2306352

[marmijo]

A workaround was added for this in: coreos/fedora-coreos-config#3127.
We're now able to run the affected kola cloud tests in rawhide and branched.