Support for WSL2 docker without Docker Desktop

[feynmanliang]

It seems that Docker Desktop and GPU passthrough are mutually exclusive. Per NVIDIA's WSL CUDA guide:

Note that NVIDIA Container Toolkit does not yet support Docker Desktop WSL 2 backend.

https://docs.nvidia.com/cuda/wsl-user-guide/index.html

To see if we can have both remote containers and GPU passthrough, I tried setting "remote.containers.dockerPath": "C:\\windows\\system32\\wsl.exe docker" but am seeing

  ERR spawn C:\Program Files\Microsoft VS Code\wsl docker ENOENT: Error: spawn C:\Program Files\Microsoft VS Code\wsl docker ENOENT
	at Process.ChildProcess._handle.onexit (internal/child_process.js:264:19)
	at onErrorNT (internal/child_process.js:456:16)
	at processTicksAndRejections (internal/process/task_queues.js:77:11)

Would it be possible to have vscode-remote directly use the underlying docker instance running within WSL2, without the dependency on Docker Desktop (or at least until Docker Desktop/NVIDIA figure out compatibility i.e. docker/roadmap#96)?

[PavelSosin-320]

I suppose it is possible and easy! I run "native docker" installed as RH CentOS platform packages using yum and CentOS documentation. inside CentOS7 distro. It doesn't contradict Cuda 11. docker-compose is there too. It has other benefits like correctly working docker CLI.
@feynmanliang If you want I can share with you tailored distro tar via OneDrive. It supports both. It is too large to share it using other methods.

[stuartleeks]

I assume that you're wanting to use devcontainers (Remote-Container) with WSL and GPU? (If not, apologies and ignore the rest of my comment!)

With the disclaimer that I haven't tried this exact scenario (and am not on the VS Code team!), I would have expected this to work without configuring the dockerPath setting if you have docker installed and in the PATH in your WSL 2 distro (and are running the code from within the distro).

When I look at the commands executed in the output for devcontainers, they all seem to be running docker via the wsl command, which seems like it would work. Of the top of my head I imagine this would require

• docker installed and in the path in the distro you want to run the container in
• your code cloned in the WSL distro that has the GPU-enabled docker instance
• you open the code via Remote-WSL initially and then choose the Re-open in container option from the menu to load the devcontainer (you should be prompted for this anyway)

[feynmanliang]

@PavelSosin-320 thank you for sharing, yes I can also confirm vscode-remote and Docker GPU passthrough work fine on linux (I'm running Arch if it matters). However, I am raising this issue specifically for the WSL2 use case.

@stuartleeks You've described exactly my use case :) I've tried "Re-open in container" without configuring dockerPath while opened in Remote-WSL, but unless I have Docker Desktop installed I get an error dialog with "Docker returned an error. Make sure the Docker daemon is running." Installing Docker Desktop resolves this error dialog, but then GPU passthrough does not work :( It seems that despite using Remote-WSL, the Remote-Container extension does not not use the docker installed inside WSL.

[Chuxel]

When I look at the commands executed in the output for devcontainers, they all seem to be running docker via the wsl command, which seems like it would work. Of the top of my head I imagine this would require

//cc: @chrmarti to confirm

[chrmarti]

We might still check with the Windows-side Docker CLI if Docker is running. I need to check and update that to use 'wsl docker' too.

[PavelSosin-320]

If Docker daemon running on the remote host it is enough to create corresponding Docker context to access it regardless of its implementation: docker running on a standalone node, Swarm or Kubernetes cluster.
When a "native" Docker daemon runs inside WSL VM it normally uses systemd unit and fd:// host URL to create a socket in the /lib/run/docker folder of the distro. This URL is the default for an automatically created context inside the distro.
But the socket itself is not a pipe used by the Windows's docker CLI implementation and is not exposed to the Windows host as WSL interop socket.
The are 2 options available: 1. Use wsl -d ... docker ... to access this "native" docker 2.install a lightweight proxy inside the distro which exposes this socket as a pipe or TCP socket.
The 1st option smoke test:
C:\Users\Pavel>wsl -d CentOS7 docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 08393e824c32 2 weeks ago 132MB
nginx 8cf1bfb43ff5 4 weeks ago 132MB
nginx alpine ecd67fe340f9 5 weeks ago 21.6MB
centos latest 831691599b88 2 months ago 215MB
alpine latest a24bb4013296 2 months ago 5.57MB
tecnativa/docker-socket-proxy latest d8b5dca2b1c8 6 months ago 20.3MB
hello-world latest bf756fb1ae65 7 months ago 13.3kB
centos/go-toolset-7-centos7 latest e772efd191e2 20 months ago 888MB

[feynmanliang]

We might still check with the Windows-side Docker CLI if Docker is running. I need to check and update that to use 'wsl docker' too.

Please let me know if there is anything I can do to support; I am not able to find the source but I am happy to validate patches.

[chrmarti]

I'm fixing the version check to run inside WSL.

The Remote Explorer viewlet is still using the local Docker and will show a message that there is no Docker when it can't find one on Windows. That shouldn't affect anything else. Keeping this issue to track that remaining work.

[chrmarti]

Fix for the version check is available in Remote-Containers 0.139.1. (Currently requires VS Code Insiders.)

[feynmanliang]

On Remote-Containers 0.139.1, I can confirm that I can now "Open in Container" to get a devcontainer running on WSL2 Ubuntu Docker. This is an improvement over erroring about Docker Desktop.

However, the "Attach Visual Studio Code" option (available under Microsoft's Docker extension tab) seems to NOOP now (previously it threw an error about Docker Desktop not installed). I suspect this feature is related to the Remote Explorer...

[chrmarti]

The default is to use the local Docker CLI. You could try opening a WSL folder (either under \\wsl$\ or with Remote-WSL) to make it use WSL's Docker CLI and then use "Attach Visual Studio Code".

How do you configure the Docker extension to work with Docker in WSL?

[feynmanliang]

Precisely as you've described above :) I open Remote-WSL Docker and the Docker extension defaults to using Docker in WSL. This is how I am reproing the behavior. On my Windows host (without Docker Desktop) the Docker extension actually raises an error and does not list WSL containers.

…

On Fri, Sep 4, 2020 at 12:58 AM Christof Marti ***@***.***> wrote: The default is to use the local Docker CLI. You could try opening a WSL folder (either under \\wsl$\ or with Remote-WSL) to make it use WSL's Docker CLI and then use "Attach Visual Studio Code". How do you configure the Docker extension to work with Docker in WSL? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3534 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHRW5LZ4HJYUFLJD3FPTMDSECM2RANCNFSM4QCYLB2A> .

[chrmarti]

Using Remote-WSL as the cue on when to use WSL's Docker CLI sounds like a better approach than just looking at the open folder. I'll look into that. 👍

[chrmarti]

Using WSL's Docker when in an empty WSL window depends on https://github.com/microsoft/vscode-remote-release/issues/3638.

The Remote-Explorer will need additional work because we want to not only show the DevContainers of the current WSL-distro and for that need to continue to hold a system-wide view. This will come up again with Remote-SSH once we get there. Removing target milestone for now.

[feynmanliang]

I see; we could possibly just use filepath / folderpath (//wsl/DistroName) to unblock this issue while #3638 lands and then later patch the detection to work for empty windows. But I'm not actively developing this feature so I could be missing some complexity here ;)

[chrmarti]

Published an improved fix with 0.148.0 of the extension (currently requires VS Code Insiders). The Remote Explorer should now also work. (Empty WSL windows still don't work.)

[feynmanliang]

Thanks, I can confirm using a Remote-WSL shows the proper containers under the "Docker" tab.

However, Remote-Containers still seems to try to default to Docker Desktop. To repro: when I try to attach to a docker container running under WSL:

I expect to be able to attach to it and view the filesystem under the explorer (possibly after selecting which WSL distro's docker to use).

Instead, I get the following error:

The "Dev Containers" terminal output suggests this is because we are trying to use Windows' docker:

[29 ms] Remote-Containers 0.148.0 in VS Code 1.51.0-insider (a5dfd9dcf83620a3b50dd56dc0117f83415c2ca8).
[84327 ms] Start: Run: docker version --format {{.Server.Version}}
[84344 ms] spawn C:\Users\Administrator\AppData\Local\Programs\Microsoft VS Code Insiders\docker ENOENT

[chrmarti]

The second screenshot in your last post shows you're not connected to WSL. Could you try attaching to the container from the Remote Explorer (or the Docker viewlet) while connected to WSL (first screenshot in your last post)?

[feynmanliang]

There is no option to attach to a running container when connected to WSL:

I tried changing the setting remote.containers.dockerPath to C:\windows\system32\wsl.exe docker but am still getting an ENOENT; it seems like this setting expects an executable rather than a CLI command w/ arguments.

[chrmarti]

You can attach to a container using the context menu on a container entry in the Docker or Remote Explorer viewlets. (Switch to 'Containers' at the top of the Remote Explorer.)

You're right, the setting needs an executable. It should work without that.

[PavelSosin-320]

The docker CLI fromChocolatery is always ready to server you Windows Docker CLI from Choco

[feynmanliang]

@chrmarti The Remote-Explorer viewletdoes not detect WSL Docker:

whereas the Docker viewlet does:

@PavelSosin-320 Thank you for the reference! Do you know how to configure the referenced Windows Docker CLI to use a WSL Docker Engine?

[PavelSosin-320]

In the regular way - using docker context create command with docker host = tcp://ip-of-machine running docker, i.e. localhost

1. On the client side create a new create a new as docker-wsl and validate it
   docker context inspect docker-wsl
   [
   {
   "Name": "docker-wsl",
   "Metadata": {
   "StackOrchestrator": "swarm"
   },
   "Endpoints": {
   "docker": {
   "Host": "tcp://localhost:2375",
   "SkipTLSVerify": false
   }
   },
   "TLSMaterial": {},
   "Storage": {
   "MetadataPath": "C:\Users\Pavel\.docker\contexts\meta\35eb697f1e12ae0856ed94eeba5d8b2f6fe336242ac93c350842ad818fb2a73d",
   "TLSPath": "C:\Users\Pavel\.docker\contexts\tls\35eb697f1e12ae0856ed94eeba5d8b2f6fe336242ac93c350842ad818fb2a73d"
   }
   }
   ]
   Use this context:
   docker context use docker-wsl.
   This context will be used by all docker clients!
   Inspect your Docker context configuration
   docker context ls
   NAME DESCRIPTION DOCKER ENDPOINT KUBERNETES ENDPOINT ORCHESTRATOR
   default * Current DOCKER_HOST based configuration npipe:////./pipe/docker_engine https://kubernetes.docker.internal:6443 (default) swarm
   docker-wsl tcp://localhost:2375 swarm

[chrmarti]

@feynmanliang Could you check with VS Code 1.51 and Remote-Containers 0.148.0? It works with these versions for me. (I'm testing by renaming the Docker executables on Windows, so they cannot be used.)

[PavelSosin-320]

On the server side I created the following daemon.json in the /etc/docker:
To satisfy WSL NIC requirement without installation additional components like socat and avoid conflicts with Docker Desktop
Anyway I stop Docker Desktop when I use this Docker engine
{
"registry-mirrors": [],
"insecure-registries": [],
"debug": true,
"ipv6": false,
"hosts": [
"tcp://localhost:2375",
"tcp://0.0.0.0:2385"
],
"data-root": "/mnt/host/d/DockerDesktopRoot",
"experimental": true
}

The main area of issues is IPV4 / IPV6 localhost interpretation in the heterogeneos Home networks - 2 localhosts per device and VM, 127.0.0.1 and [::1]
The entry point of the Cuda installation documentation is Cuda for WSL

[dparker2]

@chrmarti I was having the exact same issues described here and can confirm that upgrading Remote-Containers to 0.148.1 fixed it for me. Specifically, the Remote Explorer now shows and allows me to connect to containers in wsl

[feynmanliang]

@dparker2 Thanks! Can confirm "Attach Shell" from Docker viewlet. Are you able to "Attach Visual Studio Code" (still alert dialog with "Docker returned and error" on Remote-Containers 0.150)?

@chrmarti On Remote-Containers 0.150 I can now see the filesystem of a WSL container under the Docker viewlet of a WSL-Remote VSCode as well as open these files:

However, I am unable to open these files in the explorer; the only way that seems to work is right-click => open inside the Docker viewlet.

[satyajitghana]

So the workaround is this: https://code.visualstudio.com/docs/containers/ssh, first setup ssh in WSL2, and then use that tutorial to connect the docker windows to the WSL2 docker host.

[PavelSosin-320]

Both Docker Desktop and my Docker engine run inside VM running inside Windows OS. The approach "Remote Docker host" is not applicable to this scenario. TCP Communication between Host and Singleton WSL VM is always secure. WSL VM has no public IP, i.e. it is not visible outside Host. So Installation of SSH server inside WSL is meaningless. In any networking scenario, very slow SSH is the last resort applicable only to slow WAN without VPN. In the LAN scenario, the correct selection of Network type in the Windows setting, the correct configuration of your router, the careful pairing of network devices provide better security without paying any performance cost. SSH performance is enough only for slow terminal communication. In WAN scenario Enterprise Docker-ready server Selinux distros like RHEL and their firewalls provide military-grade communication security. over TLS 1.3 hardware-implemented encoded connection.

…

On Fri, Nov 27, 2020 at 5:37 PM Satyajit Ghana ***@***.***> wrote: So the workaround is this: https://code.visualstudio.com/docs/containers/ssh, first setup ssh in WSL2, and then use that tutorial to connect the docker windows to the WSL2 docker host. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3534 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM7YVKBKUCQSVQXRZ4OQBFDSR7BVLANCNFSM4QCYLB2A> .

[chrmarti]

@feynmanliang Could you try attaching from the Remote-Explorer viewlet? The Docker viewlet is from the Docker extension and recently added a way to browse the files inside a container (independent of Remote-Containers and not with integration into the DevContainer's tools). The likely cause of Attach Visual Studio Code from the Docker viewlet not working is microsoft/vscode#111238. That action should fall back to showing a list of running containers in a QuickPick at the top of the window, I need to investigate why it instead ends with an error for you.

@satyajitghana Connecting to a container running in WSL without having the Docker CLI installed in Windows should work now. The remaining limitations are that you need to use the Remote-Explorer for attaching (microsoft/vscode#111238) and that you need to have a folder open in WSL (microsoft/vscode#111371).

[satyajitghana]

@chrmarti Thanks ! Yes, Attach to Container Works, but then a slight inconvenience is that i have to create the volume, something the extension does automatically in Docker for Windows. But yeah, GPU is working ! so that's there.

[chrmarti]

@satyajitghana Using a devcontainer.json should also work. Do you seen any issue with that?

[satyajitghana]

@satyajitghana Using a devcontainer.json should also work. Do you seen any issue with that?

yes, that does not work for me

open folder -> Remote-WSL: Reopen folder in WSL -> Remote-Containers: Reopen in Container

produces:

[4 ms] Remote-Containers 0.148.1 in VS Code 1.51.1 (e5a624b788d92b8d34d1392e4c4d9789406efe8f).
[26 ms] Start: Resolving remote
[28 ms] Setting up container for folder or workspace: d:\Projects\app

[30 ms] Start: Check Docker is running
[30 ms] Start: Run: docker info
[116 ms] spawn d:\Projects\app\docker ENOENT
[121 ms] Docker returned an error code ENOENT, message: spawn d:\Projects\app\docker ENOENT

what is working:
start the image in Run Interactive using the Docker viewlet and then on the Containers section, Attach Shell and Attach VS Code is working for me

i just tried it, the above thing is also giving me an error now:

[6 ms] Remote-Containers 0.148.1 in VS Code 1.51.1 (e5a624b788d92b8d34d1392e4c4d9789406efe8f).
[21 ms] Start: Resolving remote
[22 ms] Setting up container /hungry_gagarin

[24 ms] Start: Run: docker inspect --type container /hungry_gagarin
[52176 ms] Start: Run: docker version --format {{.Server.Version}}
[52201 ms] spawn C:\Users\shadowleaf\AppData\Local\Programs\Microsoft VS Code\docker ENOENT

cannot attach to the container with name/id /hungry_gagarin, it no longer exists.

i think i messed up something for the attach vscode, it was definitely working before.

Attach shell is working though !

UPDATE: i tried with vscode insiders with Remote-Container v0.151.0, but same issue as above

[PavelSosin-320]

To mount bind volume to docker container the host path should be /mnt/c/....
In all cases when running containers on remote engines Docker recommends using Docker volumes with the local driver. In this case, Docker volume is created in the Docker tree:
I defined the Docker root as /mnt/host/d/DockerDesktop and docker volume create -driver local localvolume creates volumes under ... /volumes ls /mnt/host/d/DockerDesktopRoot/volumes/localvolume/_data in the same VM or Host.
This folder is not accessible from the Host side but persistent and will be recreated after the Docker daemon restart in the same place.

[chrmarti]

@satyajitghana Make sure you have a WSL folder open when you try to attach to a container. Same when reopening a folder in a container, the log shows d:\Projects\app which is a Windows folder.

The recommended way is to have Docker Desktop for Windows installed, that will also work with Windows folders. This issue is about having Docker installed only in WSL which is the case @feynmanliang is working with.

[satyajitghana]

@chrmarti yes ! that got it working, simply moving my project to WSL folder, and opening it as container, devcontainer.json is picked up as well !

[PavelSosin-320]

@satyajitghana Instead of moving your project you can point to your workspace from wsl:
/mnt/host/c/Users/MyWorkspace/MyProject -> c:\Users\MyWorkspace\MyProject and use /mnt/host/c/Users/MyWorkspace/MyProject and use it as a path to Project. Docker run -v will take it as the host path for the bind volume.
The following will work for you if you start open Linux distro in WT staying inyour project root, i.e. echo $PWD shows
/mnt/c/Users/Your-User/Profile dir:
[root@MSI myWorkspace]# cd /
[root@MSI /]# sudo mkdir /sys/fs/cgroup/systemd
[root@MSI /]# sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd
sudo docker -H localhost:2375 run -dt --mount type=bind,source=$PWD/myWorkspace/myProj,target="/myProj"
5970eb459c6c61850dfea6b3ec8746bb392f905ee64bf3693552c134386cf1e1

[feynmanliang]

ker extension and recently added a way to browse the files inside a container (independent of Remote-Containers and not with inte

@chrmarti:

@feynmanliang Could you try attaching from the Remote-Explorer viewlet?

No containers are listed under Remote-Explorer "Containers" (which I'm guessing is trying to find a docker.exe on Windows):

In contrast, the Docker extension (which I guess in this Remote-WSL VSCode is falling back on WSL2 Docker) is showing them:

[feynmanliang]

OK, I am able to get this working following @PavelSosin-320's suggestions to connect a Windows docker client over tcp to a wsl docker engine, specifically:

• Support for WSL2 docker without Docker Desktop #3534 (comment)
• Support for WSL2 docker without Docker Desktop #3534 (comment)

In addition, I needed to point the remote.containers.dockerPath setting at the chocolatey docker.exe binary.

We can close this issue.

[chrmarti]

Ok, thanks @feynmanliang.