Docker images have certain layers stuck in retrying during pull

[Taha-Firoz]

Description

My gitea setup has nginx in front of it but that cannot be the problem because it only happens to a specific layer in a docker image.

We have about 20 docker images in our registry with none of them having any problems, however 2 docker images have 1 layer that gets stuck in retrying.

First I build my image

Sending build context to Docker daemon  2.599MB
Step 1/5 : FROM node:14.11.0-buster
14.11.0-buster: Pulling from library/node
57df1a1f1ad8: Pull complete 
71e126169501: Pull complete 
1af28a55c3f3: Pull complete 
03f1c9932170: Pull complete 
65b3db15f518: Pull complete 
b071d08ac61b: Pull complete 
b3cf1273eaaa: Pull complete 
48e29c1d7527: Pull complete 
b2f0f717169a: Pull complete 
Digest: sha256:353153b8cefc14cdc9f8d0dc570c23351bd1af98d5b1593081b2ede25b3f5473
Status: Downloaded newer image for node:14.11.0-buster
 ---> 13b0e0e7fa1a
Step 2/5 : WORKDIR /app
 ---> Running in 35272956b0ad
Removing intermediate container 35272956b0ad
 ---> daff3d7c1c61
Step 3/5 : COPY . .
 ---> e9f21ed2d29e
Step 4/5 : RUN npm i
 ---> Running in 3fb0cb9e2cf4
.
.
.

then everything for this specific image gets pushed properly

562d0984732b: Pushed 
fb6ea5a74371: Pushed 
da9ca4d9f7bc: Pushed 
3b3656db1c0c: Pushed 
c35b972c0410: Pushed 
e4f96eeac2b8: Pushed 
d90cecd962d7: Pushed 
a995c5106335: Pushed 
17bdf5e22660: Pushed 
d37096232ed8: Pushed 
6add0d2b5482: Pushed 
4ef54afed780: Pushed 
latest: digest: sha256:2de0858f49e405ebe5a321a7897e8858d0383bb25559f80401155c12e6967f71 size: 2842

Then I delete all of the local images:

Untagged: gitea.local/team/node_project:latest
Untagged: gitea.local/team/node_project@sha256:2de0858f49e405ebe5a321a7897e8858d0383bb25559f80401155c12e6967f71
Deleted: sha256:9f8d7364ca1d77ee96d81376c9d01e300fb1bca8f1c05a7d822103320f7d9db1
Deleted: sha256:be72ac39787b442b484905723ecae4d3a4fcdf13e07c75f83f42a8965f218ee1
Deleted: sha256:93302413621023224753f614324e624719ce76c4d247d198f03758edfcd451c5
Deleted: sha256:e9f21ed2d29e6718508fa2971f27f8056ae2837fcc6ba1ece7183020102a0ac3
Deleted: sha256:751daa46c7a0fc6721d4aa557c4cdf573e39be49c353529868cfb9d346f8d5a0
Deleted: sha256:daff3d7c1c6102a086e0673a5289aec0439b513be7c972a63afd36bbe6f55670
Deleted: sha256:d1dd36199d9698c2192e9ec96cd92721ba628e5f56578a7735af47c79f3b7f80
Untagged: node:14.11.0-buster
Untagged: node@sha256:353153b8cefc14cdc9f8d0dc570c23351bd1af98d5b1593081b2ede25b3f5473
Deleted: sha256:13b0e0e7fa1a82780d35553a26d741be26a3b6047efd0532a83688bde22a69c3
Deleted: sha256:d0810f402a2cd03449789afaec23913af6c41575a472e371a69575d73981e755
Deleted: sha256:fe86db20a0f5e2a071373073d43a567cc412ad5183f4907201eaf8bb1208e06c
Deleted: sha256:11756f6e6fd809c8bf3476de1b4a382bc9ade261425d4a1acd1c10d054dce279
Deleted: sha256:65e4048658ae77dccb10881848e2eaefbacedb3b81d2a40fe27ed7e4f03a61d4
Deleted: sha256:c54fd0b092afa958671934519fc3c33ea83096094e3683b63b27cbf92ece78a6
Deleted: sha256:e40886afa47f1a991f807ca03358366b86c1f65471c6c7e10187cf94eae5bcd1
Deleted: sha256:28189e70cc47eb3c32769ee0aad10e6b18a8886206db4df9ccfacb50878ca1ed
Deleted: sha256:92b912e56b356aeb8e2ab873bdf557ee80fbd3bec4482684fa0f0b0c30896acc
Deleted: sha256:4ef54afed7804a44fdeb8cf6562c2a1eb745dcaabd38b1ac60126f0966bf6aef
Error: No such image: be72ac39787b
Error: No such image: e9f21ed2d29e
Error: No such image: daff3d7c1c61

and I finally pull the image from the registry

latest: Pulling from team/node_project
57df1a1f1ad8: Pull complete 
71e126169501: Pull complete 
1af28a55c3f3: Pull complete 
03f1c9932170: Download complete 
65b3db15f518: Download complete 
b071d08ac61b: Download complete 
b3cf1273eaaa: Download complete 
48e29c1d7527: Download complete 
b2f0f717169a: Retrying in 12 seconds 
edb5f27aa0e7: Download complete 
8dfde3c50bbc: Download complete 
acb701f9f05d: Download complete 

The layer b2f0f717169a gets stuck retrying indefinitely, I've tried the following:

• Deleting the package and pushing it again
• Disable proxy_buffering & proxy_request_buffering in nginx
• Upgrading my gitea from v1.7.0 to v1.7.1

We have many images in our team and all of them work fine, this one used to work fine too but for some reason this missing layer issue comes up if I make a few changes in my project and push the image again.

Gitea Version

1.17.1

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

https://gist.github.com/Taha-Firoz/8c0a6b9825879917283d47c4ee23d55c

Screenshots

No response

Git Version

No response

Operating System

ubuntu server 22.04

How are you running Gitea?

docker-compose file with mysql as it's db

Database

MySQL

[Taha-Firoz]

I don't have the logs but I can see the problem, database entries aren't verified correctly. When I push the image there are a series of head requests for each layer and only the ones that return a 404 are uploaded. This layer is incorrectly reported as existing, the Head request returns a 200 even though the layer doesn't exist in the fs. I think just adding an additional lookup to the fs wouldn't slow down the entire process a lot and would be a sure way of ensuring this bug come up.

[Taha-Firoz]

Duplicate of #19586 tracking there.