Docker CLI Windows and long paths (260 characters) not working

[kekru]

Hi there,

I think there is an issue with long paths files (longer than 260 characters), that are copied with COPY command within a Dockerfile.
Its something like moby/moby#15775, but now related to Buildkit.

Issue

This is what I did in Powershell:

# Create a dir
mkdir C:/project/aaaaaaaaaBaaaaaaaaaCaaaaaaaaaDaaaaaaaaaEaaaaaaaaaFaaaaaaaaaGaaaaaaaaaHaaaaaaaaaIaaaaaaaaaJaaaaaaaaaKaaaaaaaaaL

# Create a file that is 278 chars long
echo "hello" > C:/project/aaaaaaaaaBaaaaaaaaaCaaaaaaaaaDaaaaaaaaaEaaaaaaaaaFaaaaaaaaaGaaaaaaaaaHaaaaaaaaaIaaaaaaaaaJaaaaaaaaaKaaaaaaaaaL/278aaaaaaaaaMaaaaaaaaaNaaaaaaaaaOaaaaaaaaaPaaaaaaaaaQaaaaaaaaaRaaaaaaaaaSaaaaaaaaaTaaaaaaaaaUaaaaaaaaaVaaaaaaaaaWaaaaaaaaaXaaaaaaaaaYaaaaaaaaaZaaaaaaaaaA.txt

# Create a file that is 258 chars long
echo "hello" > C:/project/aaaaaaaaaBaaaaaaaaaCaaaaaaaaaDaaaaaaaaaEaaaaaaaaaFaaaaaaaaaGaaaaaaaaaHaaaaaaaaaIaaaaaaaaaJaaaaaaaaaKaaaaaaaaaL/258aaaaaaaaaMaaaaaaaaaNaaaaaaaaaOaaaaaaaaaPaaaaaaaaaQaaaaaaaaaRaaaaaaaaaSaaaaaaaaaTaaaaaaaaaUaaaaaaaaaVaaaaaaaaaWaaaaaaaaaXaaaaaaaaa.txt

Having following Dockerfile

FROM alpine:3.9.4
COPY . /
RUN ls -la /aaaaaaaaaBaaaaaaaaaCaaaaaaaaaDaaaaaaaaaEaaaaaaaaaFaaaaaaaaaGaaaaaaaaaHaaaaaaaaaIaaaaaaaaaJaaaaaaaaaKaaaaaaaaaL

Running without Buildkit both files are inside the image, as expected:

PS C:\project> Remove-Item env:DOCKER_BUILDKIT

PS C:\project> docker build --progress plain --no-cache .
Sending build context to Docker daemon   7.68kB
Step 1/3 : FROM alpine:3.9.4
 ---> 055936d39205
Step 2/3 : COPY . /
 ---> a289f56da592
Step 3/3 : RUN ls -la /aaaaaaaaaBaaaaaaaaaCaaaaaaaaaDaaaaaaaaaEaaaaaaaaaFaaaaaaaaaGaaaaaaaaaHaaaaaaaaaIaaaaaaaaaJaaaaaaaaaKaaaaaaaaaL
 ---> Running in 847e5d70224f
total 16
drwxr-xr-x    2 root     root          4096 Feb 17 17:37 .
drwxr-xr-x    1 root     root          4096 Feb 17 17:53 ..
-rwxr-xr-x    1 root     root            16 Feb 17 17:37 258aaaaaaaaaMaaaaaaaaaNaaaaaaaaaOaaaaaaaaaPaaaaaaaaaQaaaaaaaaaRaaaaaaaaaSaaaaaaaaaTaaaaaaaaaUaaaaaaaaaVaaaaaaaaaWaaaaaaaaaXaaaaaaaaa.txt
-rwxr-xr-x    1 root     root            16 Feb 17 17:35 278aaaaaaaaaMaaaaaaaaaNaaaaaaaaaOaaaaaaaaaPaaaaaaaaaQaaaaaaaaaRaaaaaaaaaSaaaaaaaaaTaaaaaaaaaUaaaaaaaaaVaaaaaaaaaWaaaaaaaaaXaaaaaaaaaYaaaaaaaaaZaaaaaaaaaA.txt
Removing intermediate container 847e5d70224f
 ---> f3984890e8e9
Successfully built f3984890e8e9
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.

Running with Buildkit only the shorter file is inside the image:

PS C:\project> $env:DOCKER_BUILDKIT=1

PS C:\project> docker build --progress plain --no-cache .
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 192B done
#1 DONE 0.4s

#2 [internal] load .dockerignore
#2 transferring context: 2B 0.0s done
#2 DONE 0.4s

#3 [internal] load metadata for docker.io/library/alpine:3.9.4
#3 DONE 0.0s

#4 [1/3] FROM docker.io/library/alpine:3.9.4
#4 CACHED

#5 [internal] load build context
#5 transferring context: 591B 0.0s done
#5 DONE 0.1s

#6 [2/3] COPY . /
#6 DONE 0.1s

#7 [3/3] RUN ls -la /aaaaaaaaaBaaaaaaaaaCaaaaaaaaaDaaaaaaaaaEaaaaaaaaaFaaaa...
#7 1.252 total 12
#7 1.252 drwxr-xr-x    2 root     root          4096 Feb 17 17:38 .
#7 1.252 drwxr-xr-x    1 root     root          4096 Feb 17 17:55 ..
#7 1.252 -rwxr-xr-x    1 root     root            16 Feb 17 17:37 258aaaaaaaaaMaaaaaaaaaNaaaaaaaaaOaaaaaaaaaPaaaaaaaaaQaaaaaaaaaRaaaaaaaaaSaaaaaaaaaTaaaaaaaaaUaaaaaaaaaVaaaaaaaaaWaaaaaaaaaXaaaaaaaaa.txt
#7 DONE 1.4s

#8 exporting to image
#8 exporting layers
#8 exporting layers 0.1s done
#8 writing image sha256:c739fecfe5648b356900207a6cb7d6672138d1ffc3dfdbf2f5251986eda451db
#8 writing image sha256:c739fecfe5648b356900207a6cb7d6672138d1ffc3dfdbf2f5251986eda451db 0.0s done
#8 DONE 0.2s

My Setup

Using current Docker Desktop for Windows:

PS C:\project> docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:37 2019
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[matthid]

I just run into this issue and it was quite frustrating to find the root cause. In my situation some random files were missing in the container and the build would fail. Like the author describes this only happened when using BuildKit.
I feel the only reason why not a lot of people have run into this is because nobody is using BuildKit on windows apparently.

[MrLuje]

Also ran into this issue... the fact that it's silently failing during copy makes it extremly hard to identify this about path being too long

[gionnyFuego]

Did you solve this issue? I notive for node container this issue.

[rhlarora84]

Ran into the same issue with Java codebase.. took a while to figure out that the compilation was happening due to missing class files with long paths !

[johannesnormannjensen]

I too have run into this issue. Is this being looked into?

[kekru]

If it is not easy to fix: It would be nice, if there were an error message, complaining about too long paths

[nzbart]

Failing silently isn't a great approach to being unable to read a file. It makes me question the integrity of the containers I build.

I worked around the problem by using the following path syntax:

docker build \\?\D:\path -f .

Mapping D:\path to \\?\D:\path allows the use of long file paths.

[Saugat168]

I too ran into the same problem and spent a few hours debugging an error message that I got at Docker run, complaining with a cryptic message that was only a side effect of this real problem. As people pointed out, silent failures definitely isn't great. Here's the link to my problem ,arising out of this , on stackoverflow: https://stackoverflow.com/questions/72604450/mlflow-load-model-fails-python/72613045#72613045

[profnandaa]

Cross-linking -> #1086

[profnandaa]

@tonistiigi -- also if possible, could we add an extra tag on top of area/windows to differentiate LCOW vs. Windows Containers?
/cc. @gabriel-samfira

[profnandaa]

Looks like this issue was resolved along the way?

Can't repro it no more. On Docker v24.0.6 (that runs Buildkit backend for Linux by default):

PS C:\dev\play\buildkit\long_path> docker build -t lcow_longpath . --progress plain --no-cache
....
#7 [3/3] RUN ls -la /aaaaaaaaaBaaaaaaaaaCaaaaaaaaaDaaaaaaaaaEaaaaaaaaaFaaaaaaaaaGaaaaaaaaaHaaaaaaaaaIaaaaaaaaaJaaaaaaaaaKaaaaaaaaaL
#7 0.302 total 12
#7 0.302 drwxr-xr-x    2 root     root          4096 Oct 19 08:52 .
#7 0.302 drwxr-xr-x    1 root     root          4096 Oct 19 08:58 ..
#7 0.302 -rwxr-xr-x    1 root     root            16 Oct 19 08:55 278aaaaaaaaaMaaaaaaaaaNaaaaaaaaaOaaaaaaaaaPaaaaaaaaaQaaaaaaaaaRaaaaaaaaaSaaaaaaaaaTaaaaaaaaaUaaaaaaaaaVaaaaaaaaaWaaaaaaaaaXaaaaaaaaaYaaaaaaaaaZaaaaaaaaaA.txt
#7 DONE 0.3s

#8 exporting to image
...

[profnandaa]

Please re-open if this is still happening but couldn't repro from my end.