Docker Image on Windows Base Image #439 #2332

jwdb · 2024-01-03T13:47:09Z

With this PR it's possible to build a Windows version of the Azurite container. I've not added the required steps to the package.json since the current steps very much look internally used and I would be unable to test them.

It is based upon the work from @shanrath in the issue (#439), expanding it with a NodeJS baseimage because of the lack of a proper nodejs windows baseimage that can be trusted to be updated regularly.

The image is based upon the LTSC 2022 version of Windows Nanoserver.

Thanks for contribution! Please go through following checklist before sending PR.

PR Branch Destination

For Azurite V3, please send PR to main branch.
For legacy Azurite V2, please send PR to legacy-dev branch.

Always Add Test Cases

Make sure test cases are added to cover the code change.

Add Change Log

Add change log for the code change in Upcoming Release section in ChangeLog.md.

Development Guideline

Please go to CONTRIBUTION.md for steps about setting up development environment and recommended Visual Studio Code extensions.

jwdb · 2024-01-03T13:48:58Z

@microsoft-github-policy-service agree

blueww · 2024-01-04T07:34:21Z

@jwdb

Thanks for the contribution!

Would you please confirm:
Would you like Azurite team to release windows base docker image, or just keep this script in Azurite to help customer create image locally?

If you would like Azurite to release windows based docker image, please also add:

Update package.json so we can build the docker image from npm command. You can refer this PR: feat: add scripts for building multi-architecture azurite docker images #1569
Add Unit test for windows base docker image build / run. You can refer:

Azurite/azure-pipelines.yml

Line 528 in 3f64420

- job: docker

If you just would like to keep this script in Azurite repo, would you please:
Give the detail steps to build windows based docker image, and better update Readme.md with it (if the step is long, you can add a doc into https://github.com/Azure/Azurite/tree/main/docs/designs, and link Readme.md with it.).

jwdb · 2024-01-04T09:10:24Z

Hi @blueww,

I would like the image to be released, that way it's easier to use & make it easier for the windows image to stay up-to-date with the latest version of Azurite.

I've added the build steps I think are neccesary to the package.json & azure-pipelines.yml.

blueww · 2024-01-04T09:39:21Z

It looks there are something wrong with the pipeline change, only 3 PR checks run, but originally there are 30+ PR checks.

blueww · 2024-01-04T09:43:22Z

package.json

    "docker:publish-arm64": "cross-var docker push xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-arm64",
-    "docker:create-manifest-versioned": "cross-var docker manifest create xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-amd64 xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-arm64",
-    "docker:create-manifest-latest": "cross-var docker manifest create xstoreazurite.azurecr.io/public/azure-storage/azurite:latest xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-amd64 xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-arm64",
+    "docker:create-manifest-versioned": "cross-var docker manifest create xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-amd64 xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-windows-amd64 xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-arm64",


Just to check what's the effect after publish the manifest.
With the originally manifest, when user run Azurite docker image without specifying platform, docker will choose the docker image most match the current platform (ARM64 platform will choose AMD64 docker image, AMD64 platform will choose AMD64 docker image). After the windows-amd64 image is added, what will happen when user run Azurite docker image without specifying platform?

The platform is chooses includes the os name, hence it currently throwing an error on windows ( no matching manifest for windows/amd64 in the manifest list entries) adding this image will add that specific platform to the manifest.

With current Azurite release (before the change), I will install AMD64 docker image in my machine (my machine is windows with AMD64).
So after your change, what will happen on different platform/OS?

And for the error, do you mean you will resolve it, or it's already resolved?

If you are running Docker in Windows Container mode (this is an option within Docker-Desktop), with the current release this will throw an error.
After the change, the other platforms will be unaffected (since they will use the linux/amd64 image).

You can view the platform docker is running internally on with docker info -f "{{.OSType}}". For me this gives windows.

Get it. Thanks!

And on my windows machine run docker info -f "{{.OSType}}", will get result "linux".

blueww · 2024-01-04T09:50:12Z

I tried to build windows based image in code space, and meet following error:

@blueww ➜ /workspaces/Azurite (main) $ npm run docker:build-windows-amd64

> azurite@3.29.0 docker:build-windows-amd64
> cross-var docker buildx build --platform windows/amd64 --load --no-cache --rm -f "Dockerfile.Windows" -t xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version-windows-amd64 .

[+] Building 0.8s (2/2) FINISHED                                                                                                                                                                                                                                                  docker:default
 => [internal] load build definition from Dockerfile.Windows                                                                                                                                                                                                                                0.4s
 => => transferring dockerfile: 1.18kB                                                                                                                                                                                                                                                      0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                           0.4s
 => => transferring context: 282B                                                                                                                                                                                                                                                           0.0s
Dockerfile.Windows:34
--------------------
  32 |     # Production image
  33 |     #
  34 | >>> FROM Node-Windows
  35 |     
  36 |     ENV NODE_ENV=productions
--------------------
ERROR: failed to solve: failed to parse stage name "Node-Windows": invalid reference format: repository name must be lowercase

Adding workdir change to node base image

jwdb · 2024-01-04T09:57:50Z

Interesting enough I do not encounter that error when running it locally.
I've changed the Dockerfile for a lowercase stage name.
Also I saw that the Azure pipeline error'ed on the curl, I've added a workdir change to hopefully mitigate that issue.

blueww · 2024-01-04T10:06:15Z

Interesting enough I do not encounter that error when running it locally. I've changed the Dockerfile for a lowercase stage name. Also I saw that the Azure pipeline error'ed on the curl, I've added a workdir change to hopefully mitigate that issue.

If possible, I would suggest you testing the build docker image step in the github code space (it's linux based), and make sure it passes.

…ows containers documentation.

jwdb · 2024-01-04T10:17:14Z

It is not possible to build or run a Windows container on a Linux machine, as far as I know there isn't a (proper) way to run Windows on a Github codespace.

blueww · 2024-01-08T05:51:04Z

Dockerfile.Windows

+FROM mcr.microsoft.com/windows/nanoserver:ltsc2022-amd64 AS nodewindows
+# Install dependencies first
+RUN mkdir c:\node
+WORKDIR c:\\node


Try to build docker image on my win10 machine failed:

D:\code\azurite>npm run docker:build-windows > azurite@3.29.0 docker:build-windows D:\code\azurite > npm run docker:prebuild && cross-var docker build --no-cache --rm -f "Dockerfile.Windows" -t xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version . && cross-var docker tag xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version xstoreazurite.azurecr.io/public/azure-storage/azurite:latest npm WARN config cache-max This option has been deprecated in favor of `--prefer-online` npm WARN config cache-min This option has been deprecated in favor of `--prefer-offline`. npm WARN config optional Use `--omit=optional` to exclude optional dependencies, or npm WARN config `--include=optional` to include them. npm WARN config npm WARN config Default value does install optional deps unless otherwise omitted. npm WARN config shrinkwrap Use the --package-lock setting instead. npm WARN config tmp This setting is no longer used. npm stores temporary files in a special npm WARN config location in the cache, and they are managed by npm WARN config [`cacache`](http://npm.im/cacache). > azurite@3.29.0 docker:prebuild D:\code\azurite > echo skip skip [+] Building 14.0s (6/23) docker:default => [internal] load build definition from Dockerfile.Windows 0.1s => => transferring dockerfile: 1.22kB 0.0s => [internal] load .dockerignore 0.1s => => transferring context: 282B 0.0s => [internal] load metadata for mcr.microsoft.com/windows/nanoserver:ltsc2022-amd64 0.9s => [nodewindows 1/6] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022-amd64@sha256:505fd350350a443863a2f8e827878216343389429572ec5e7b89aab4d18dce10 12.7s => => resolve mcr.microsoft.com/windows/nanoserver:ltsc2022-amd64@sha256:505fd350350a443863a2f8e827878216343389429572ec5e7b89aab4d18dce10 0.0s => => sha256:505fd350350a443863a2f8e827878216343389429572ec5e7b89aab4d18dce10 429B / 429B 0.0s => => sha256:46a42778ee1d5da936c5a65c66528eb6c727cd190a7935fe60cb33017bd29f45 638B / 638B 0.0s => => sha256:3221147efd2bf92b8a3ff6e6b8206dfab879e886d1b97b07845f9eb5ddd0ab50 116.99MB / 116.99MB 8.7s => => extracting sha256:3221147efd2bf92b8a3ff6e6b8206dfab879e886d1b97b07845f9eb5ddd0ab50 3.9s => [internal] load build context 5.2s => => transferring context: 23.58MB 5.2s => ERROR [nodewindows 2/6] RUN mkdir c:node 0.2s ------ > [nodewindows 2/6] RUN mkdir c:node: ------ Dockerfile.Windows:6 -------------------- 4 | FROM mcr.microsoft.com/windows/nanoserver:ltsc2022-amd64 AS nodewindows 5 | # Install dependencies first 6 | >>> RUN mkdir c:\node 7 | WORKDIR c:\\node 8 | -------------------- ERROR: failed to solve: process "cmd /S /C mkdir c:\\node" did not complete successfully: unable to find user ContainerUser: invalid argument npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! azurite@3.29.0 docker:build-windows: `npm run docker:prebuild && cross-var docker build --no-cache --rm -f "Dockerfile.Windows" -t xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version . && cross-var docker tag xstoreazurite.azurecr.io/public/azure-storage/azurite:$npm_package_version xstoreazurite.azurecr.io/public/azure-storage/azurite:latest` npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the azurite@3.29.0 docker:build-windows script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above. npm ERR! A complete log of this run can be found in: npm ERR! C:\Users\weiwei\AppData\Roaming\npm-cache\_logs\2024-01-08T05_44_24_027Z-debug.log

My machine has:
Runtime Environment:
OS Name: Windows
OS Version: 10.0.19045
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\7.0.404\

What type of containers are you running? The error seems to be related with running Linux Containers instead of Windows Containers. (You can check with docker info -f "{{.OSType}}" and switch using the context menu of docker desktop Switch to Windows containers).

I have switched docker on my machine to Windows.
However, run npm run docker:build-windows will still fail.
See failure log:
2024-01-08T09_30_37_462Z-debug.log

I do not see the actual exception in the logfile. From what I see one of the child processes spawned by the npm command died with an error, do you have a log of the subcommand that failed?
Running the command with the same node & npm versions as specified in the logfile does not trigger any error at my side (nor on DevOps).

After I upgrade node to LTS version, and try again, I meet following error:

ltsc2022-amd64: Pulling from windows/nanoserver
a Windows version 10.0.20348-based image is incompatible with a 10.0.19045 host

It looks my windows version is lower than the image and cause the docker image build failure.
I will try to upgrade my machine or find a new machine and try build it. However, it might need some time.
Will update you later.

I can successfully build the docker image on a win11 machine with node 18.19.0.
However, get following warning.
So to get the new docker image in good shape, we should increasing node version.

npm WARN notsup Unsupported engine for tedious@16.6.1: wanted: {"node":">=16"} (current: {"node":"14.21.3","npm":"6.14.18"}) npm WARN notsup Not compatible with your version of node/npm: tedious@16.6.1 npm verb notsup Not compatible with your version of node/npm: tedious@16.6.1 npm verb notsup Required: {"node":">=16"} npm verb notsup Actual: {"npm":"6.14.18","node":"14.21.3"} npm WARN notsup Unsupported engine for @azure/core-util@1.6.1: wanted: {"node":">=16.0.0"} (current: {"node":"14.21.3","npm":"6.14.18"}) npm WARN notsup Not compatible with your version of node/npm: @azure/core-util@1.6.1 npm verb notsup Not compatible with your version of node/npm: @azure/core-util@1.6.1 npm verb notsup Required: {"node":">=16.0.0"} npm verb notsup Actual: {"npm":"6.14.18","node":"14.21.3"} npm WARN notsup Unsupported engine for lru-cache@8.0.5: wanted: {"node":">=16.14"} (current: {"node":"14.21.3","npm":"6.14.18"}) npm WARN notsup Not compatible with your version of node/npm: lru-cache@8.0.5 npm verb notsup Not compatible with your version of node/npm: lru-cache@8.0.5 npm verb notsup Required: {"node":">=16.14"} npm verb notsup Actual: {"npm":"6.14.18","node":"14.21.3"} npm WARN notsup Unsupported engine for @azure/core-rest-pipeline@1.13.0: wanted: {"node":">=18.0.0"} (current: {"node":"14.21.3","npm":"6.14.18"}) npm WARN notsup Not compatible with your version of node/npm: @azure/core-rest-pipeline@1.13.0 npm verb notsup Not compatible with your version of node/npm: @azure/core-rest-pipeline@1.13.0 npm verb notsup Required: {"node":">=18.0.0"} npm verb notsup Actual: {"npm":"6.14.18","node":"14.21.3"} npm WARN notsup Unsupported engine for @azure/msal-node@2.6.1: wanted: {"node":">=16"} (current: {"node":"14.21.3","npm":"6.14.18"}) npm WARN notsup Not compatible with your version of node/npm: @azure/msal-node@2.6.1 npm verb notsup Not compatible with your version of node/npm: @azure/msal-node@2.6.1 npm verb notsup Required: {"node":">=16"} npm verb notsup Actual: {"npm":"6.14.18","node":"14.21.3"}

blueww · 2024-01-08T05:51:49Z

Dockerfile.Windows

+RUN tar -xf Node.zip -C c:\node
+RUN del Node.zip
+
+ENV PATH="$PATH;C:\Node\node-v14.21.3-win-x64"


Could we use node with a higher version?

We could, but then this Dockerfile would deviate from the linux containers version (which also is using Node 14). Also the build process would be slightly different (options like npm config set unsafe-perm=true are no longer available in Node 16+).

To me it feels like upgrading the node version should be a seperate issue, since the current Dockerfile also uses Node 14 and there will be changes needed in the build commands.

For the new added docker image, hope it's adding with a good shape. So the node version should be in the recommended version list (better with LTS version): https://nodered.org/docs/faq/node-versions

I've just tried updating to a newer version of Node (16/18/20) and it seems like I'm hitting npm/cli#4027. The current prepare script executes tsc which is unavailable and unwanted during the install phase. Changing this would require either a rather dirty workaround (setting the prepare step to "" in the docker file) or changes in the normal build process specified in the package.json. If desired I could implement these changes, but it feels like it's going rather out of scope.

Thanks for the update!

Yes, for the new added docker image, we would like it be in the good shape.
It would be great if you could add the change to make it on the node LTS version.

Dockerfile for Windows container

3e8584d

blueww self-requested a review January 4, 2024 07:47

blueww self-assigned this Jan 4, 2024

jwdb added 2 commits January 4, 2024 10:06

Add build scripts to package.json

c79f97f

Add build step for windows docker container

f1488ab

blueww reviewed Jan 4, 2024

View reviewed changes

Rename jobname because the name has to be a letter, number or underscore

51de33f

changing the Node-Windows stage name to nodewindows.

729f43e

Adding workdir change to node base image

Add cmd to run command to validate docker image in line with the wind…

0ac3aff

…ows containers documentation.

blueww reviewed Jan 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker Image on Windows Base Image #439 #2332

Docker Image on Windows Base Image #439 #2332

jwdb commented Jan 3, 2024 •

edited

Loading

jwdb commented Jan 3, 2024

blueww commented Jan 4, 2024 •

edited

Loading

jwdb commented Jan 4, 2024

blueww commented Jan 4, 2024

blueww Jan 4, 2024

jwdb Jan 4, 2024

blueww Jan 4, 2024

jwdb Jan 4, 2024

blueww Jan 4, 2024 •

edited

Loading

blueww commented Jan 4, 2024 •

edited

Loading

jwdb commented Jan 4, 2024

blueww commented Jan 4, 2024 •

edited

Loading

jwdb commented Jan 4, 2024

blueww Jan 8, 2024

jwdb Jan 8, 2024

blueww Jan 8, 2024

jwdb Jan 8, 2024

blueww Jan 9, 2024 •

edited

Loading

blueww Jan 10, 2024

blueww Jan 8, 2024

jwdb Jan 8, 2024 •

edited

Loading

blueww Jan 8, 2024 •

edited

Loading

jwdb Jan 8, 2024

blueww Jan 9, 2024

Docker Image on Windows Base Image #439 #2332

Are you sure you want to change the base?

Docker Image on Windows Base Image #439 #2332

Conversation

jwdb commented Jan 3, 2024 • edited Loading

PR Branch Destination

Always Add Test Cases

Add Change Log

Development Guideline

jwdb commented Jan 3, 2024

blueww commented Jan 4, 2024 • edited Loading

jwdb commented Jan 4, 2024

blueww commented Jan 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blueww Jan 4, 2024 • edited Loading

Choose a reason for hiding this comment

blueww commented Jan 4, 2024 • edited Loading

jwdb commented Jan 4, 2024

blueww commented Jan 4, 2024 • edited Loading

jwdb commented Jan 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blueww Jan 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jwdb Jan 8, 2024 • edited Loading

Choose a reason for hiding this comment

blueww Jan 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jwdb commented Jan 3, 2024 •

edited

Loading

blueww commented Jan 4, 2024 •

edited

Loading

blueww Jan 4, 2024 •

edited

Loading

blueww commented Jan 4, 2024 •

edited

Loading

blueww commented Jan 4, 2024 •

edited

Loading

blueww Jan 9, 2024 •

edited

Loading

jwdb Jan 8, 2024 •

edited

Loading

blueww Jan 8, 2024 •

edited

Loading