podman rm --storage fails

[banool]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I have a container that I cannot start because it says the name is already in use, but I can't delete the resources related to the name.

Steps to reproduce the issue:
Try to start container, see that it fails:

$ /usr/bin/podman run   -a stdout -a stderr   --cgroups no-conmon   --conmon-pidfile %t/%n-pid   --cidfile %t/%n-cid --name diary-django quay.io/banool/diary-django:latest
Error: error creating container storage: the container name "diary-django" is already in use by "1cad8118630c7559567d960f11a1236e82726d962b4f633b5ee4d3930b2a6c28". You have to remove that container to be able to reuse that name.: that name is already in use

The container isn't actually there, but try to remove it:

$ /usr/bin/podman rm -f --storage diary-django
Error: error unmounting container "diary-django": layer not known

$ podman rm --storage 1cad8118630c7559567d960f11a1236e82726d962b4f633b5ee4d3930b2a6c28
Error: error looking up container "1cad8118630c7559567d960f11a1236e82726d962b4f633b5ee4d3930b2a6c28" mounts: layer not known

Describe the results you received:
See above.

Describe the results you expected:
I'd expect the cleanup to work.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:      3.2.3
API Version:  3.2.3
Go Version:   go1.15.13
Built:        Sat Aug  7 10:21:14 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.21.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.29-1.el8.3.9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 2
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: journald
  hostname: littlesally
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.18.0-305.10.2.el8_4.x86_64
  linkmode: dynamic
  memFree: 535928832
  memTotal: 3788779520
  ociRuntime:
    name: crun
    package: crun-0.20.1-1.el8.3.6.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.20.1
      commit: 0d42f1109fd73548f44b01b3e84d04a279e99d2e
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-4.el8.7.13.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 4102025216
  swapTotal: 4102025216
  uptime: 9m 14.38s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /home/daniel/.config/containers/storage.conf
  containerStore:
    number: 12
    paused: 0
    running: 10
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.el8.5.3.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.5
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/daniel/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 21
  runRoot: /run/user/1000
  volumePath: /home/daniel/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.3
  Built: 1628356874
  BuiltTime: Sat Aug  7 10:21:14 2021
  GitCommit: ""
  GoVersion: go1.15.13
  OsArch: linux/amd64
  Version: 3.2.3

Package info (e.g. output of rpm -q podman or apt list podman):

$ rpm -q podman
podman-3.2.3-1.el8.1.5.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes, this issue isn't mentioned.

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical CentOS 8 machine.

I tried restarting the machine to no avail.

[mheon]

Can you do a podman ps -a --external and see if the container is showed in the output?

[banool]

$ podman ps -a --external | grep diary
1cad8118630c  quay.io/banool/diary-django:latest                                         storage               7 weeks ago   storage                                      diary-django

Would you look at that. What should I do about this from here? I'm not quite sure what --external means.

Update, I see this: http://docs.podman.io/en/latest/markdown/podman-ps.1.html. So I figure podman rm cannot remove these images that are stored in some external place, so I have to remove them manually some other way? Would it make sense for podman to have the ability to clean these up or would that be out of scope for it.

[mheon]

@nalind PTAL - I think the error messages here are coming out of storage.

[nalind]

It looks like libpod.Runtime.removeStorageContainer() is checking if store.Mounted() returns storage.ErrContainerUnknown for the supplied ID, but I don't think it ever returns that error. Mounted() generally expects a layer ID, but internally looks up the one for a container if it's given a valid container ID. It returns ErrLayerUnknown if the container's layer appears to have been removed.

[mheon]

Ack. I'm assuming that error should be non-fatal, and proceeding will successfully remove the image?

[nalind]

I would expect DeleteContainer() using that ID to then succeed or return a multi-error, or ErrNotAContainer.

[banool]

In the interim, is there a workaround? I'm happy to nuke all my images / containers if need be.

[banool]

For anyone else who is having this issue, I ran this:

buildah unshare
rm -rf .local/share/containers/

This completely nukes every container and image, so use this with extreme caution, but it fixed my issue. If you're stumbling across this further into the future, you may not want this, since the underlying issue might be fixed by then.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@nalind @mheon Are you suggesting that we do

diff --git a/libpod/runtime_cstorage.go b/libpod/runtime_cstorage.go
index cd2f226af..995a46397 100644
--- a/libpod/runtime_cstorage.go
+++ b/libpod/runtime_cstorage.go
@@ -112,12 +112,14 @@ func (r *Runtime) removeStorageContainer(idOrName string, force bool) error {
                        return errors.Wrapf(define.ErrCtrStateInvalid, "container %q is mounted and cannot be removed without using force", idOrName)
                }
        } else if _, err := r.store.Unmount(ctr.ID, true); err != nil {
-               if errors.Cause(err) == storage.ErrContainerUnknown {
+               if errors.Is(err, storage.ErrContainerUnknown) {
                        // Container again gone, no error
                        logrus.Infof("Storage for container %s already removed", ctr.ID)
                        return nil
                }
-               return errors.Wrapf(err, "error unmounting container %q", idOrName)
+               if !errors.Is(err, storage.ErrLayerUnknown) {
+                       return errors.Wrapf(err, "error unmounting container %q", idOrName)
+               }
        }
 
        if err := r.store.DeleteContainer(ctr.ID); err != nil {

[nalind]

@rhatdan checking for storage.ErrLayerUnknown is what I'd recommend, yes, but I don't think errors wrapped using github.com/pkg/errors implement an Is() method, so using errors.Is() instead of comparing using errors.Cause() doesn't look like it'll work.

[mheon]

@Luap99 noted that this should work - we've been using it in all newly-landed PRs for Podman.

[nalind]

Oh, you're right, it implements Unwrap().