c9s: afterburn hitting selinux denials when installing an OKD cluster

[Prashanth684]

When installing an OKD cluster, some nodes do not come up. It turns out they do not have a node name because the afterburn service does not run. It errors out due to selinux denials:

[core@ip-10-0-29-129 ~]$ systemctl status afterburn.service
× afterburn.service - Afterburn (Metadata)
     Loaded: loaded (/usr/lib/systemd/system/afterburn.service; disabled; preset: disabled)
     Active: failed (Result: exit-code) since Thu 2024-07-18 06:32:31 UTC; 11h ago
       Docs: https://coreos.github.io/afterburn/usage/attributes/
   Main PID: 879 (code=exited, status=1/FAILURE)
        CPU: 42ms

Jul 18 06:32:30 localhost afterburn[879]: Jul 18 06:32:30.747 INFO Putting http://169.254.169.254/latest/api/token: Attempt #1
Jul 18 06:32:31 ip-10-0-29-129 afterburn[879]: Jul 18 06:32:31.765 INFO Putting http://169.254.169.254/latest/api/token: Attempt #2
Jul 18 06:32:31 ip-10-0-29-129 afterburn[879]: Error: failed to run
Jul 18 06:32:31 ip-10-0-29-129 afterburn[879]: Caused by:
Jul 18 06:32:31 ip-10-0-29-129 afterburn[879]:     0: writing metadata attributes
Jul 18 06:32:31 ip-10-0-29-129 afterburn[879]:     1: failed to create directory "/run/metadata"
Jul 18 06:32:31 ip-10-0-29-129 afterburn[879]:     2: Permission denied (os error 13)
Jul 18 06:32:31 ip-10-0-29-129 systemd[1]: afterburn.service: Main process exited, code=exited, status=1/FAILURE
Jul 18 06:32:31 ip-10-0-29-129 systemd[1]: afterburn.service: Failed with result 'exit-code'.
Jul 18 06:32:31 ip-10-0-29-129 systemd[1]: Failed to start Afterburn (Metadata).

Also, the denials in the audit logs:

time->Thu Jul 18 17:41:27 2024
type=PROCTITLE msg=audit(1721324487.450:7946): proctitle=2F7573722F62696E2F61667465726275726E002D2D636D646C696E65002D2D617474726962757465733D2F72756E2F6D657461646174612F61667465726275726E
type=PATH msg=audit(1721324487.450:7946): item=1 name=(null) inode=6892 dev=00:18 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:var_run_t:s0 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1721324487.450:7946): item=0 name=(null) inode=1 dev=00:18 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:var_run_t:s0 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1721324487.450:7946): cwd="/"
type=SYSCALL msg=audit(1721324487.450:7946): arch=c000003e syscall=83 success=yes exit=0 a0=7ffd8fe9d9d0 a1=1ff a2=e a3=5635a2531097 items=2 ppid=1 pid=6572 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="afterburn" exe="/usr/bin/afterburn" subj=system_u:system_r:afterburn_t:s0 key=(null)
type=AVC msg=audit(1721324487.450:7946): avc:  denied  { create } for  pid=6572 comm="afterburn" name="metadata" scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1721324487.450:7946): avc:  denied  { add_name } for  pid=6572 comm="afterburn" name="metadata" scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1721324487.450:7946): avc:  denied  { write } for  pid=6572 comm="afterburn" name="/" dev="tmpfs" ino=1 scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=dir permissive=1
----
time->Thu Jul 18 17:41:27 2024
type=PROCTITLE msg=audit(1721324487.450:7947): proctitle=2F7573722F62696E2F61667465726275726E002D2D636D646C696E65002D2D617474726962757465733D2F72756E2F6D657461646174612F61667465726275726E
type=PATH msg=audit(1721324487.450:7947): item=3 name=(null) inode=6893 dev=00:18 mode=0100644 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:var_run_t:s0 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1721324487.450:7947): item=2 name=(null) inode=6892 dev=00:18 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:var_run_t:s0 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1721324487.450:7947): item=1 name=(null) nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1721324487.450:7947): item=0 name=(null) inode=6892 dev=00:18 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:var_run_t:s0 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1721324487.450:7947): cwd="/"
type=SYSCALL msg=audit(1721324487.450:7947): arch=c000003e syscall=257 success=yes exit=6 a0=ffffff9c a1=7ffd8fe9d9d8 a2=80241 a3=1b6 items=4 ppid=1 pid=6572 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="afterburn" exe="/usr/bin/afterburn" subj=system_u:system_r:afterburn_t:s0 key=(null)
type=AVC msg=audit(1721324487.450:7947): avc:  denied  { write open } for  pid=6572 comm="afterburn" path="/run/metadata/afterburn" dev="tmpfs" ino=6893 scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=1
type=AVC msg=audit(1721324487.450:7947): avc:  denied  { create } for  pid=6572 comm="afterburn" name="afterburn" scontext=system_u:system_r:afterburn_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=1
----
time->Thu Jul 18 17:42:45 2024
type=PROCTITLE msg=audit(1721324565.288:8006): proctitle=2F7573722F62696E2F61667465726275726E002D2D636D646C696E65002D2D7373682D6B6579733D636F7265
type=PATH msg=audit(1721324565.288:8006): item=0 name="/var/home/core/.ssh/authorized_keys.d/" inode=17825920 dev=103:04 mode=040700 ouid=1000 ogid=1000 rdev=00:00 obj=unconfined_u:object_r:ssh_home_t:s0 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1721324565.288:8006): cwd="/"
type=SYSCALL msg=audit(1721324565.288:8006): arch=c000003e syscall=87 success=no exit=-2 a0=7fff03f1d1f8 a1=7fff03f1d1f8 a2=30 a3=55f52afed66a items=1 ppid=1 pid=6648 auid=4294967295 uid=0 gid=0 euid=1000 suid=0 fsuid=1000 egid=1000 sgid=0 fsgid=1000 tty=(none) ses=4294967295 comm="afterburn" exe="/usr/bin/afterburn" subj=system_u:system_r:afterburn_t:s0 key=(null)
type=AVC msg=audit(1721324565.288:8006): avc:  denied  { search } for  pid=6648 comm="afterburn" name=".ssh" dev="nvme0n1p4" ino=16777344 scontext=system_u:system_r:afterburn_t:s0 tcontext=unconfined_u:object_r:ssh_home_t:s0 tclass=dir permissive=1
----
time->Thu Jul 18 17:42:45 2024
type=PROCTITLE msg=audit(1721324565.288:8007): proctitle=2F7573722F62696E2F61667465726275726E002D2D636D646C696E65002D2D7373682D6B6579733D636F7265
type=SYSCALL msg=audit(1721324565.288:8007): arch=c000003e syscall=257 success=yes exit=6 a0=ffffff9c a1=7fff03f1d1e8 a2=80000 a3=0 items=0 ppid=1 pid=6648 auid=4294967295 uid=0 gid=0 euid=1000 suid=0 fsuid=1000 egid=1000 sgid=0 fsgid=1000 tty=(none) ses=4294967295 comm="afterburn" exe="/usr/bin/afterburn" subj=system_u:system_r:afterburn_t:s0 key=(null)
type=AVC msg=audit(1721324565.288:8007): avc:  denied  { open } for  pid=6648 comm="afterburn" path="/var/home/core/.ssh/authorized_keys.d" dev="nvme0n1p4" ino=17825920 scontext=system_u:system_r:afterburn_t:s0 tcontext=unconfined_u:object_r:ssh_home_t:s0 tclass=dir permissive=1
type=AVC msg=audit(1721324565.288:8007): avc:  denied  { read } for  pid=6648 comm="afterburn" name="authorized_keys.d" dev="nvme0n1p4" ino=17825920 scontext=system_u:system_r:afterburn_t:s0 tcontext=unconfined_u:object_r:ssh_home_t:s0 tclass=dir permissive=1

This has started happening after #1552 where we had to use selinux version selinux-policy-38.1.36-1.el9 as selinux-policy-38.1.36-1.el9 is not available anymore (#1514).

[jlebon]

Can you file a bug on the RHEL board against the selinux-policy component and the version set to CentOS Stream 9?

[Prashanth684]

@aleskandro already filed: https://issues.redhat.com/browse/RHEL-49735