/etc/hosts does not contain container name

[ngarratt]

We're migrating some workloads from an old Nomad/Docker setup to Nomad/Podman, but seeing a mismatch between jobs invoked directly by 'podman run' and those invoked by Nomad. Both are being tested in bridge mode to the same cni-podman0 bridge.

Podman is setting the following which all work fine:

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
<bridge_ip> host.containers.internal host.docker.internal
<container_ip> <container_id> <container_name>

Nomad is setting the following

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
<host_ip> host.containers.internal host.docker.internal

Setting the host.containers.internal to the host IP is causing some introspection issues when doing reverse lookup against that IP, and no entry for <container_id> causes different Java binding failures because it can't resolve the hostname to an IP.

Is there a setting in the driver (or Nomad) that influences this behavior?

OS: RHEL 8
Nomad: 1.8.1
Nomad CNI: 1.5.1
Podman: 4.9.4
nomad-driver-podman: 0.6.1

[ngarratt]

This looks to be an issue with Nomad invoking CNI directly instead of allowing podman to handle networking.

The podman/libnetwork code that populates /etc/hosts is making decisions based on 'network.Driver'. If the Nomad task is not specifying a network mode, it defaults to "nsmode":"bridge" when invoking against libpod. That works fine.

But if the Nomad task has a network mode specified, it engages CNI to build the network then uses "nsmode":"path" pointing to the netns from the CNI when invoking against libpod. This turns off all the etchost logic in podman/libnetwork for bridge networks.

In our case we can just work around it by not specifying a network mode for bridge networks, and by modifying the /etc/hosts file in our host network containers during prestart. This podman driver isn't directly building the /etc/hosts file, I'm not sure what the correct solution is.