Podman do not create working directory if it do not exist when using the Docker REST API

[mscherer]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When using podman compatiblity layer (eg, the podman-docker rpm installed), woodpecker is unable to start a container, while it work fine with Docker.

Woodpecker is a go based CI system that do everything in a container. For now, it only support Docker, but I tried to test with Podman.

Steps to reproduce the issue:

1. deploy woodpecker-server and agent

2. install podman-docker

3. do a test build

Describe the results you received:

Upon doing a test build, woodpecker will create a container to do the git checkout, and that fail at that step with this error message in the log:

workdir "/drone/src/git.example.org/misc/blog" does not exist on container 56ed6ac93e0d828e90ea0bd1ca4cadad65d04a9afef200fa8bc0af9e880f8666

upon further debugging, this is the call to /v1.33/containers/0_5095234068184889173_clone_0/start that fail (error 500 + that error)

The container was created before with a call to /v1.33/containers/create with notably:

• working dir set to "/drone/src/git.example.org/misc/blog"

• the volume /drone mounted (json extract: "Volumes:{"/drone":{} }"

(I have the full json if needed)

Since /drone is a new volume, it will be empty. Since it will be empty, the subdirectory do not exist, so it fail.

Describe the results you expected:

I forced podman to create the workingdir with some creative editing around the code that print the error message on https://github.com/containers/podman/blob/main/libpod/container_internal_linux.go#L210 and it worked.

From a quick search, it seems moby create the working directory unconditionally:
https://github.com/moby/moby/blob/master/container/container.go#L266

Podman/libpod create only if a option (libpod.WithCreateWorkingDir) is passed when creating the container.

And while there is maybe some good reasons, I would expect the compat API to mirror docker/moby behavior.

Additional information you deem important (e.g. issue happens only occasionally):

This was first discussed on woodpecker-ci/woodpecker#85

Output of podman version:

Version:      3.4.0
API Version:  3.4.0
Go Version:   go1.16.8
Built:        Thu Sep 30 21:40:21 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.30-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 2
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: wikibase.example.org
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.12.12-300.fc34.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 567738368
  memTotal: 4041846784
  ociRuntime:
    name: runc
    package: runc-1.0.2-2.fc34.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.2
      commit: e15f155-dirty
      spec: 1.0.2-dev
      go: go1.16.6
      libseccomp: 2.5.0
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc34.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 8210345984
  swapTotal: 8212443136
  uptime: 1803h 32m 11.22s (Approximately 75.12 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 3
    stopped: 6
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageStore:
    number: 7
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.0
  Built: 1633030821
  BuiltTime: Thu Sep 30 21:40:21 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.4.0-1.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

F34 VM, on a RHEL 7 host.

[mheon]

Any chance you can provide the exact call Compose is making to create the container?

[mscherer]

That's not using Docker Compose, that's directly the agent using the API. But I can try to get a tcpdump with the data removed (making a reproducer shouldn't be too hard on test infra).

Would the raw tcpdump file be fine ?

[mheon]

Yes, that'd work fine

[mscherer]

Here is the dump (using socat + tcpdump).

woodpecker.dump.gz

Wireshark open it without issue (on a recent one, eg, not the one on RHEL 7 like I tried first). The whole json should be inside, as well as answers.

[mheon]

OK, I think the issue here is that the workdir is located on a volume. I recall merging a fix for this for Podman 3.4.0; any chance you can retest with that?

[mheon]

Wait, I need to read more - you're already on 3.4.0. Damn.

[rhatdan]

If Workdir is on a volume, it could be the chown change that matches the volume to the destdir rather then the user of the image.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Since I got no response to my idea, I am closing. If you want to respond and see if this is the problem then reopen.

[mscherer]

Why is the bug closed ? There was no sign of it being fixed, and and I gave all information. Should I repopen a bug and link that one ?

[Footur]

If Workdir is on a volume, it could be the chown change that matches the volume to the destdir rather then the user of the image.

@mscherer, can you please respond to rhatdan's comment.

[rhatdan]

If you respond then we can continue the conversation. We have hundreds of issues, and when reporters do not responde for 30 days I close the issue.

[mscherer]

Well, I can answer, but I do not understand the question, as it seems not related to the problem from my perspective.

The volume /drone is created by podman, as the API call do not have a src directory in argument, cf my 1st comment. From what I understand, that mean that podman will create it on the fly in /var/lib/containers/

The container is started with Workdir set to a subdirectory of that volume, who was just created empty by podman.

On Docker/Moby, the engine create the subdirectory unconditionnaly (cf the link I gave).
On podman using the podman official API, this seems to requires a flag ( CreateWorkingDir).
On podman with the docker compat API, this flag is not set, so the directory is not created, and it fail with the error message in the 1st comment.

Forcing the configuration (c.config.CreateWorkingDir) on https://github.com/containers/podman/blob/main/libpod/container_internal_linux.go#L203 make it work.
By forcing, I mean to recompile podman after commenting that part of the code so podman do not verify c.config.CreateWorkingDir. Since the check is no longer here, podman create the directory, and it work.

Since everything is done by podman, I am not sure exactly which chown the suggestion refer to, as there is 18 occurrences in the container_internal_linux.go file. I am not sure exactly what I should test in the first place, because it fail around line 208 and the nearest call to chown is around line 228. resolveWorkDir (from container_internal_linux.go ).

Since the problem can be worked around by removing a check so podman match the behavior of docker/moby, I think the problem is not a permission issue, but a problem of API mismatch.

[rhatdan]

In the compat API we should just force the CreateWorkingDir option. please open a PR for that.

[mscherer]

Done. So far, I tested by replacing podman on my personal CI, but I will also add it to ./python/docker/compat/ once I have a grasp of the current testing infra.