Invalid cross-device link with named volume

[giuseppe]

Copied from: coreos/fedora-coreos-tracker#1168

Describe the bug
Running a container in podman on fcos next, fails with an exdev error when it tries to rename a file mounted under a named volume to something in the same directory.

Reproduction steps
Steps to reproduce the behavior:

sudo podman run -v "redpanda-test:/var/lib/redpanda/data" docker.io/vectorized/redpanda:v21.11.12
the first run, it will start ok, you should see a log line with "Successfully started Redpanda!"
the second run, the container exits with Failure during startup: std::__1::__fs::filesystem::filesystem_error (error system:18, filesystem error: rename failed: Invalid cross-device link [/var/lib/redpanda/data/redpanda/kvstore/0_0/snapshot.partial.1649900837775.a1ZO] [/var/lib/redpanda/data/redpanda/kvstore/0_0/snapshot])
Expected behavior
Container should restart without issue, being able to rename the file.

Actual behavior
Container dies with an odd error

System details

• Bare Metal/QEMU/AWS/GCP/etc. Proxmox VM

• Fedora CoreOS version next (v 36.20220410.1.1)

Ignition config
very sparse, just ssh keys, networkmanager configs (would be sanitized) and podman.socket.

Additional information
The container starts without issue if I point the directory to a bind mount instead of a named volume. I can only reproduce this on fcos, not on Arch Linux with podman 4.0.3, kernel 5.17.1 or centos 8 stream with podman 4.0.2.

[dustymabe]

regarding the proposed -o o=noquota solution.. Is it possible to get around this without a special setup beforehand? i.e. If I'm starting a container that's using a volume that has no other owners then re-use the projid or something?

disclaimer: I know very little about XFS. I'm just asking questions.

[giuseppe]

regarding the proposed -o o=noquota solution.. Is it possible to get around this without a special setup beforehand? i.e. If I'm starting a container that's using a volume that has no other owners then re-use the projid or something?

doable but it is more complex. We would need to check that the projid is not used by another container and this would work only when volumes are using the same projid, as well as make sure the quota settings are the same. The real difficulty is that the volume management is done afterwards, so we would need to change a few things before that.

In any case it can be done as a follow up improvement

[nivekuil]

Should volumes default to noquota for docker compatibility?

[giuseppe]

not sure, that looks more like a missing feature in Docker to not take advantage of quotas to track how much is used by a container