CNI: Make firewalld rules --permanent

[Schrottfresse]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I configured CNI to use firewalld as firewall backend and added my podman network interface to the trusted zone. When I'm reloading firewalld (firewall-cmd --reload) all my podman related rules go missing. I'd like to have the ability to make the podman firewalld rules --permanent.
(I'm starting my containers as root.)

Steps to reproduce the issue:

1. Start container

2. firewall-cmd --reload

3. Rules are missing.

Describe the results you received:

# podman start pihole
5d95bb331f8d59c4d7d5da9af98570b3f82dd933aaf22126650e31741f225a6a
# podman ps
CONTAINER ID  IMAGE                                   COMMAND  CREATED            STATUS             PORTS  NAMES
5d95bb331f8d  docker.io/pihole/pihole:latest                   About an hour ago  Up 49 seconds ago         pihole
# firewall-cmd --zone=trusted --list-all
trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces:
  sources: 172.16.21.4/32 fd00:c001::9e/128
  services:
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
# firewall-cmd --reload
success
# firewall-cmd --zone=trusted --list-all
trusted
  target: ACCEPT
  icmp-block-inversion: no
  interfaces:
  sources:
  services:
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Describe the results you expected:

Rules don't go missing.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

# podman version
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.12.12
OS/Arch:            linux/amd64

Output of podman info --debug:

# podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.12.12
  podman version: 1.6.4
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.6-1.module_el8.1.0+272+3e64ee36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.6, commit: 7a4f0dd7b20a3d4bf9ef3e5cbfac05606b08eac0'
  Distribution:
    distribution: '"centos"'
    version: "8"
  MemFree: 32140668928
  MemTotal: 33369112576
  OCIRuntime:
    name: runc
    package: runc-1.0.0-64.rc9.module_el8.1.0+272+3e64ee36.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 16835932160
  SwapTotal: 16835932160
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: carter
  kernel: 4.18.0-147.5.1.el8_1.x86_64
  os: linux
  rootless: false
  uptime: 1h 35m 6.12s (Approximately 0.04 days)
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - registry.fedoraproject.org
  - registry.centos.org
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 2
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 3
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

# rpm -q podman
podman-1.6.4-2.module_el8.1.0+272+3e64ee36.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

physical server

[mheon]

I don't think we can do this because we want the rules to disappear after reboot.

We've talked about this before, and the best we think we can do is offer a command that forces network reconfiguration for all running containers, recreating the rules.

[baude]

@mheon doesnt this also make some assumptions about the network stack being exactly the same each time as well? Given that we have zero interaction with firewalld itself, this seems nearly impossible.

would it be possible to have something like a run-hook option where an executable could be run once the container state hit running or some such? of course, this could be done nicely with systemd as well and is maybe the more appropriate approach.

[mheon]

If we do it any later, we end up with a race where traffic after the container started but before the network started will be lost.

We can probably preserves IP address and MAC address on container recreation, even for non-static IP and MAC (we have a mechanism built-in already for doing this, originally for checkpoint/restore). I think we can probably duplicate the network configuration "well enough" with this (it works for checkpoint/restore on a different system, which seems promising)

[ttys3]

I've encountered with this problem too.

sudo podman run --name ngxdemo -p 8086:80 nginxdemos/hello

❯ sudo podman inspect ngxdemo | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "10.88.1.5",

❯ sudo firewall-cmd --zone=trusted --list-all 
trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: 
  sources: 10.88.1.2/32 10.88.1.1/32 10.88.1.4/32 10.88.1.5/32
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

reload firewalld rules:

sudo firewall-cmd --reload

sources empty now:

❯ sudo firewall-cmd --zone=trusted --list-all                  
trusted
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

try restart one of the container:

❯ sudo podman restart ngxdemo
Error: cannot listen on the TCP port: listen tcp4 :8086: bind: address already in use

try restart again, this time it will success:

❯ sudo podman restart ngxdemo
c7a0efe0c7dc540717c301216f01fef7f9eba12315418c16ed554dd8733e2add

and got back:

❯ sudo podman inspect ngxdemo | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "10.88.1.6",

❯ sudo firewall-cmd --zone=trusted --list-all  
trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: 
  sources: 10.88.1.4/32 10.88.1.1/32 10.88.1.6/32
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

---

ps: I even use firewalld as CNI firewall plugin backend, not the default iptables.

❯ cat /etc/cni/net.d/87-podman-bridge.conflist
{
    "cniVersion": "0.4.0",
    "name": "podman",
    "plugins": [
	{
            "type": "bridge",
            "bridge": "cni-podman0",
            "isGateway": true,
            "ipMasq": true,
            "ipam": {
		"type": "host-local",
		"routes": [
		    {
			"dst": "0.0.0.0/0"
		    }
		],
		"ranges": [
		    [
			{
			    "subnet": "10.88.0.0/16",
			    "gateway": "10.88.0.1"
			}
		    ]
		]
            }
	},
	{
            "type": "portmap",
            "capabilities": {
		"portMappings": true
            }
	},
	{
            "type": "firewall",
	    "backend": "firewalld"
	}
    ]
}

[abalage]

Although this is really annoying, as I recall Docker + firewalld worked the very same way.
I always had to restart docker daemon to recreate its iptables chains and rules after I did a 'firewall-cmd --reload'. I might recall that containers had to be redeployed as well.

For me what worked is to use a container specific OS (like OpenSUSE MicroOS, Fedora CoreOS) explicitly dedicated for containers and leave network and firewall management to podman alone. I can use firewall rules outside of this VM.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon Any movement on this one?

[mheon]

No. I do want to add a command to do this, but there is no way we'll have time until Podmanv2 is done.

[KlausThornProgrammfabrik]

To my understanding, firewalld and nftables were designed to solve this exact type of problem:

• does not assume complete control of firewall backend
• won’t delete firewall rules installed by other tools or users

... from: https://developers.redhat.com/blog/2018/08/10/firewalld-the-future-is-nftables/

I guess podman/CNI would have to use nftables and create own tables. And leave firewalld to the user's local customization, only. Then firewalld-cmd --reload will just affect the user's stuff.

Have you investigated this? Are there plans for this?

[dcode]

I just ran into this as well and thinking through it, it makes sense. I'm not familiar enough with the intimate details, but seems like CNI doesn't have any sort of daemon (sort of the point, I guess), would it be possible to use a dbus-activated service (via systemd) that could examine the state of containers and recreate the appropriate entries when firewalld reloads, e.g. when Reloaded signal is emitted on org.fedoraproject.FirewallD1

[sergeiwaigant]

Same here with Podman 1.6.4 on RHEL7.7 EUS.
I don't understand why this issue is not being tackled.
With this situation we would need to disable Firewalld at all to use Podman containers as services - or is there any other workaround to overcome that problem?

We are using systems maintained by central team with Puppet and they run firewalld --reload every 30 minutes, this is breaking the iptables and the services in the containers are not working anymore.

[mheon]

As said before, --permanent is not an option because it has undesirable side effects - we do not want rules to outlive the container that created them, as would be the case after reboots. We are considering a podman system command that will recreate rules after a firewall reload, which I suppose you could run after your firewalld --reload though there would be an interruption in connectivity.

Can I ask why you are reloading firewalld on such a regular basis? I don't think I've heard of anyone doing that before, and it's definitely not something I think we (or the CNI maintainers) have considered.

[sergeiwaigant]

OK. Let's say it differently.
Any reload, which can happen from tme to time even manualy, should not result in any kind of unavailablity of a service running in a container.
If that should be normal, then any kind of system services running in container are just useless.

[EssGeeEich]

What about adding a --persistent flag for the podman run command?

[mheon]

That doesn't solve the fundamental problem that the rules will survive reboot, and the CNI plugins do not deal well with reboots - we'll leak rules if any container is still running when the system is shut down, and that will break things badly when the IP in question is reused for a fresh container. Podman itself does not directly create the rules, so I can't hack around this during our usual post-reboot cleanup because I don't know what rules need to be changed. The CNI plugins are theoretically safe to call on an already-dead container after a reboot to finalize cleanup of things like these firewall rules, but this has never actually worked for us.

[rhatdan]

@mccv1r0 @mheon Any new ideas on this one? Seems to be an issue with CNI Plugins.

[Masber]

as a workaround, I managed to go over this issue by moving Centos 8 firewall configuration from nftables to iptables

[mnsmithuk]

I found the best workaround for now was to run :

firewall-cmd --permanent --zone=trusted --add-interface=cni-podman0

but not sure if this introduces a security issue in anyway.

[mnsmithuk]

My bad. It does not work because podman populates trusted sources not trusted interfaces. On a reboot the source disappears and the conatiner is unreachable. I thought the workaround I suggested had fixed this yesterday but just confirmed it is actually still in the same position.

[thkukuk]

Can I ask why you are reloading firewalld on such a regular basis?

I see the same problem that the podman rules go away regular. The reason for the firewalld reload here are intrusion detection tools, which reconfigure the firewall according to entries they find in the logs.

[mchouque]

Regarding the reload thing, on Fedora, NetworkManager has this postinstall line:

test -f /usr/bin/firewall-cmd && firewall-cmd --reload --quiet || :

Meaning when you upgrade the package, you lose rules created by podman.

[jwhonce]

For work-around/current state please see, 29-container-related-firewall-rules-are-lost-after-reloading-firewalld

[githubcek]

Nice solution. On "Red Hat Enterprise Linux release 8.5 (Ootpa)" solution with 2 .service files is working for firewall-cmd --reload and systemctl restart firewalld but not for: systemctl stop firewalld and then systemctl start firewalld
I have deleted "Requires=firewalld.service" and now also works for systemctl start firewalld :)

cat /etc/systemd/system/podman-firewall-cmd-reload.service
my /bash is on different location so I had to change it a little:

[Unit]
Description=firewalld reload hook - run a hook script on firewalld reload
Wants=dbus.service
After=dbus.service

[Service]
Type=simple
ExecStart=/usr/bin/bash -c '/usr/bin/busctl monitor --system --match "interface=org.fedoraproject.FirewallD1,member=Reloaded" --match "interface=org.fedoraproject.FirewallD1,member=PropertiesChanged" | while read -r line ; do podman network reload --all ; done'

[Install]
WantedBy=multi-user.target

cat /etc/systemd/system/podman-firewalld-restart.service
Here I had to change " and ' characters otherwise it didn't work:

[Unit]
Description=Redo podman NAT rules after firewalld starts or reloads
Wants=dbus.service
After=dbus.service

[Service]
Type=simple
ExecStart=/usr/bin/bash -c '/usr/bin/dbus-monitor --profile --system "type=signal,sender=org.freedesktop.DBus,path=/org/freedesktop/DBus,interface=org.freedesktop.DBus,member=NameAcquired,arg0=org.fedoraproject.FirewallD1" "type=signal,path=/org/fedoraproject/FirewallD1,interface=org.fedoraproject.FirewallD1,member=Reloaded" | sed -u "/^#/d" | while read -r type timestamp serial sender destination path interface member _junk; do if [[ $type = "#"* ]]; then continue; elif [[ $interface = org.freedesktop.DBus && $member = NameAcquired ]]; then echo "firewalld started"; podman network reload --all; elif [[ $interface = org.fedoraproject.FirewallD1 && $member = Reloaded ]]; then echo "firewalld reloaded"; podman network reload --all; fi; done'
Restart=Always

[Install]
WantedBy=default.target

systemctl daemon-reload
systemctl start podman-firewall-cmd-reload
systemctl start podman-firewalld-restart
systemctl enable podman-firewall-cmd-reload
systemctl enable podman-firewalld-restart

Thank you all.

[rhatdan]

@edsantiago PTAL
@mheon Is this something we should ship?

[mheon]

I'm not opposed, though I'm still in favor of integrating this into Aardvark at some point in the future (4.1?) so we can automatically handle the reload there, no extra systemd service required.

[edsantiago]

Sorry, I don't think I can review this: I don't know much about firewalld, and know even less about dbus.

[rhatdan]

@githubcek Please open a PR to add you firewall script, we can ship this with Podman and not enable it by default, then we can ask people to try it out.

[yrro]

On my systems I've been using a single service to react to firewalld startup and reload events. This service also parses the output of dbus-monitor to ensure that podman network reload --all is only run once per event.

(I think I put this into another similar bug already, but don't have the reference handy, sorry about that).

This seems to work fine for me but I think condensing it into a single linen shell script is... not great. A short Python script would be much clearer. If you'd accept it in Podman then I can work on a PR--let me know.

(I forget why I didn't use busctl monitor -- either at the time it needed a feature that RHEL's busctl lacked, or maybe it was easier to parse the output of dbus-montor?)

# /etc/systemd/system/podman-firewalld-reload.service
[Unit]
Description=Redo podman NAT rules after firewalld starts or reloads
Wants=dbus.service
After=dbus.service

[Service]
Type=simple
Environment=LC_CTYPE=C.utf8
ExecStart=/bin/bash -c "dbus-monitor --profile --system 'type=signal,sender=org.freedesktop.DBus,path=/org/freedesktop/DBus,interface=org.freedesktop.DBus,member=NameAcquired,arg0=org.fedoraproject.FirewallD1' 'type=signal,path=/org/fedoraproject/FirewallD1,interface=org.fedoraproject.FirewallD1,member=Reloaded' | sed -u '/^#/d' | while read -r type timestamp serial sender destination path interface member _junk; do if [[ $type = '#'* ]]; then continue; elif [[ $interface = org.freedesktop.DBus && $member = NameAcquired ]]; then echo 'firewalld started'; podman network reload --all; elif [[ $interface = org.fedoraproject.FirewallD1 && $member = Reloaded ]]; then echo 'firewalld reloaded'; podman network reload --all; fi; done"
Restart=always
#ExecStart=/bin/bash -c "busctl monitor --system --match type=signal,sender=org.freedesktop.DBus,path=/org/freedesktop/DBus,interface=org.freedesktop.DBus,member=NameAcquired,arg0=org.fedoraproject.FirewallD1 --match type=signal,path=/org/fedoraproject/FirewallD1,interface=org.fedoraproject.FirewallD1,member=Reloaded | while read line; do if [[ $line = ‣* ]]; then podman network reload --all; fi; echo "$line"; done"

[Install]
WantedBy=multi-user.target

[jameslikeslinux]

I've been waiting for a one-file, no-script workaround to this issue before making the switch to firewalld and @yrro's service has been working well for me. Thanks! Hope to see a proper solution integrated soon.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Not likely to happen on CNI, but maybe netavark.

[jritter]

I just ran into this issue on RHEL8, podman network reload --all helped. Is there a solution on the horizon for this?

[mheon]

Aside from the linked systemd/scripting workaround, there are no immediate plans, though we have ideas about integrating a DBus listener into Aardvark to handle reload events.

[jritter]

Thanks @mheon, we'll keep using the workaround for now then.

[danishprakash]

While work on this is going on, is it okay if we add the systemd unit file within podman.spec?

[Luap99]

This is fixed with netavark v1.9 and a new systemd service: https://blog.podman.io/2023/11/new-netavark-firewalld-reload-service/

Thus I am going to close this issue, we will not fix this for CNI.