Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Force removing image with 2 tags and running container does not remove the container #7153

Closed
marusak opened this issue Jul 30, 2020 · 3 comments · Fixed by #7155
Closed

Force removing image with 2 tags and running container does not remove the container #7153

marusak opened this issue Jul 30, 2020 · 3 comments · Fixed by #7155
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@marusak
Copy link
Contributor

marusak commented Jul 30, 2020

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Force removing image with 2 tags and running container does not remove the container.
I created simple script, see below. The first rmi call fails, calling it again succeeds, removes both images but keeps the container running. The container cannot be then inspected nor attached to, but is still shown as running.
If I don't tag the container (removing the third line in the script) the first removal succeeds and all is as expected.

Steps to reproduce the issue:

$ cat script1.sh 
podman pull docker.io/library/fedora:latest
id=`podman inspect --format '{{.Id}}' docker.io/library/fedora:latest`
podman tag docker.io/library/fedora:latest docker.io/library/fedora:32
podman run -dt docker.io/library/fedora:latest bash

podman rmi -f $id
sleep 5
podman rmi -f $id

podman images -a
podman ps -a

$ sudo podman ps -a
CONTAINER ID  IMAGE                 COMMAND  CREATED      STATUS   PORTS                                           NAMES
597b71ce3dab  k8s.gcr.io/pause:3.1           4 weeks ago  Created  0.0.0.0:80->5000/tcp, 0.0.0.0:27017->27017/tcp  31f255dd9f21-infra

$ sudo podman images -a
REPOSITORY               TAG     IMAGE ID      CREATED      SIZE
docker.io/library/mongo  latest  2b2cc1f48aed  6 weeks ago  392 MB
k8s.gcr.io/pause         3.1     da86e6ba6ca1  2 years ago  749 kB

$ sudo ./script1.sh
Trying to pull docker.io/library/fedora:latest...
Getting image source signatures
Copying blob c7def56d621e done  
Copying config a368cbcfa6 done  
Writing manifest to image destination
Storing signatures
a368cbcfa6789bc347345f6d19132afe138b62ff5373d2aa5f37120277c90b54
3fd9dfa4eda0dacc53d4120d4e6a97c784ead84d4a27a3bcb24c4208a7c1e6aa
Error: 1 error occurred:
	* error removing containers [e3f852c3b57d68017969518e70c4a8dd558da0ef5d11a0f75e43eced8e9aa098] for image "a368cbcfa6789bc347345f6d19132afe138b62ff5373d2aa5f37120277c90b54": could not remove container "e3f852c3b57d68017969518e70c4a8dd558da0ef5d11a0f75e43eced8e9aa098": 2 errors occurred:
	* unlinkat /var/lib/containers/storage/overlay-containers/e3f852c3b57d68017969518e70c4a8dd558da0ef5d11a0f75e43eced8e9aa098/userdata/shm: device or resource busy
	* unlinkat /var/lib/containers/storage/overlay-containers/e3f852c3b57d68017969518e70c4a8dd558da0ef5d11a0f75e43eced8e9aa098/userdata/shm: device or resource busy
Untagged: docker.io/library/fedora:latest
Untagged: docker.io/library/fedora:32
Deleted: a368cbcfa6789bc347345f6d19132afe138b62ff5373d2aa5f37120277c90b54
REPOSITORY               TAG     IMAGE ID      CREATED      SIZE
docker.io/library/mongo  latest  2b2cc1f48aed  6 weeks ago  392 MB
k8s.gcr.io/pause         3.1     da86e6ba6ca1  2 years ago  749 kB
CONTAINER ID  IMAGE                            COMMAND  CREATED        STATUS            PORTS                                           NAMES
3fd9dfa4eda0  docker.io/library/fedora:latest  bash     5 seconds ago  Up 5 seconds ago                                                  admiring_golick
597b71ce3dab  k8s.gcr.io/pause:3.1                      4 weeks ago    Created           0.0.0.0:80->5000/tcp, 0.0.0.0:27017->27017/tcp  31f255dd9f21-infra

Describe the results you received:
Container is still running even though its image is not present.

Describe the results you expected:
Container would be deleted the same way it gets deleted when the image has only one tag.

Output of podman version:

Version:      2.0.2
API Version:  1
Go Version:   go1.14.3
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

NOTE:
I noticed this when I was force deleting through APIv2. There it should behave the same.
@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 30, 2020
@mheon
Copy link
Member

mheon commented Jul 30, 2020

Reproduces. The container has definitely been removed from storage as part of the removal of the image, so the inconsistency seems to be that we removed it directly from c/storage and not via Podman's removal functions.

@mheon
Copy link
Member

mheon commented Jul 30, 2020

Think I found it. Simple logic error wasn't invoking container delete, so the c/storage code kicked in.

mheon added a commit to mheon/libpod that referenced this issue Jul 30, 2020
@mheon
Ensure that 'rmi --force' evicts Podman containers
adc3301 
The logic for `podman rmi --force` includes a bit of code that
will remove Libpod containers using Libpod's container removal
logic - this ensures that they're cleanly and completely removed.
For other containers (Buildah, CRI-O, etc) we fall back to
manually removing the containers using the image from c/storage.

Unfortunately, our logic for invoking the Podman removal function
had an error, and it did not properly handle cases where we were
force-removing an image with >1 name. Force-removing such images
by ID guarantees their removal, not just an untag of a single
name; our code for identifying whether to remove containers did
not proper detect this case, so we fell through and deleted the
Podman containers as storage containers, leaving traces of them
in the Libpod DB.

Fixes containers#7153

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
@mheon mheon mentioned this issue Jul 30, 2020
Merged
@mheon
Copy link
Member

mheon commented Jul 30, 2020

Fix in #7155

mheon added a commit to mheon/libpod that referenced this issue Jul 30, 2020
@mheon
Ensure that 'rmi --force' evicts Podman containers
55d39cf 
The logic for `podman rmi --force` includes a bit of code that
will remove Libpod containers using Libpod's container removal
logic - this ensures that they're cleanly and completely removed.
For other containers (Buildah, CRI-O, etc) we fall back to
manually removing the containers using the image from c/storage.

Unfortunately, our logic for invoking the Podman removal function
had an error, and it did not properly handle cases where we were
force-removing an image with >1 name. Force-removing such images
by ID guarantees their removal, not just an untag of a single
name; our code for identifying whether to remove containers did
not proper detect this case, so we fell through and deleted the
Podman containers as storage containers, leaving traces of them
in the Libpod DB.

Fixes containers#7153

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
mheon added a commit to mheon/libpod that referenced this issue Jul 30, 2020
@mheon
Ensure that 'rmi --force' evicts Podman containers
9f5a11c 
The logic for `podman rmi --force` includes a bit of code that
will remove Libpod containers using Libpod's container removal
logic - this ensures that they're cleanly and completely removed.
For other containers (Buildah, CRI-O, etc) we fall back to
manually removing the containers using the image from c/storage.

Unfortunately, our logic for invoking the Podman removal function
had an error, and it did not properly handle cases where we were
force-removing an image with >1 name. Force-removing such images
by ID guarantees their removal, not just an untag of a single
name; our code for identifying whether to remove containers did
not proper detect this case, so we fell through and deleted the
Podman containers as storage containers, leaving traces of them
in the Libpod DB.

Fixes containers#7153

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
mheon added a commit to mheon/libpod that referenced this issue Jul 31, 2020
@mheon
Ensure that 'rmi --force' evicts Podman containers
8e97245 
The logic for `podman rmi --force` includes a bit of code that
will remove Libpod containers using Libpod's container removal
logic - this ensures that they're cleanly and completely removed.
For other containers (Buildah, CRI-O, etc) we fall back to
manually removing the containers using the image from c/storage.

Unfortunately, our logic for invoking the Podman removal function
had an error, and it did not properly handle cases where we were
force-removing an image with >1 name. Force-removing such images
by ID guarantees their removal, not just an untag of a single
name; our code for identifying whether to remove containers did
not proper detect this case, so we fell through and deleted the
Podman containers as storage containers, leaving traces of them
in the Libpod DB.

Fixes containers#7153

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
mheon added a commit to mheon/libpod that referenced this issue Aug 17, 2020
@mheon
Ensure that 'rmi --force' evicts Podman containers
d1065f4 
The logic for `podman rmi --force` includes a bit of code that
will remove Libpod containers using Libpod's container removal
logic - this ensures that they're cleanly and completely removed.
For other containers (Buildah, CRI-O, etc) we fall back to
manually removing the containers using the image from c/storage.

Unfortunately, our logic for invoking the Podman removal function
had an error, and it did not properly handle cases where we were
force-removing an image with >1 name. Force-removing such images
by ID guarantees their removal, not just an untag of a single
name; our code for identifying whether to remove containers did
not proper detect this case, so we fell through and deleted the
Podman containers as storage containers, leaving traces of them
in the Libpod DB.

Fixes containers#7153

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
mheon added a commit to mheon/libpod that referenced this issue Aug 17, 2020
@mheon
Ensure that 'rmi --force' evicts Podman containers
c109ca2 
The logic for `podman rmi --force` includes a bit of code that
will remove Libpod containers using Libpod's container removal
logic - this ensures that they're cleanly and completely removed.
For other containers (Buildah, CRI-O, etc) we fall back to
manually removing the containers using the image from c/storage.

Unfortunately, our logic for invoking the Podman removal function
had an error, and it did not properly handle cases where we were
force-removing an image with >1 name. Force-removing such images
by ID guarantees their removal, not just an untag of a single
name; our code for identifying whether to remove containers did
not proper detect this case, so we fell through and deleted the
Podman containers as storage containers, leaving traces of them
in the Libpod DB.

Fixes containers#7153

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Ensure that 'rmi --force' evicts Podman containers mheon/libpod
3 participants
@mheon @marusak @openshift-ci-robot