Any plans for an ARM64-compatible release

[andyraf]

Are there any plans to support an ARM64-compatible release of Git for Windows?

[PhilipOakley]

My ignorance here, but is Windows on ARM64?

This maybe an 'upstream' question to see if the new Azure CI capability could build and test for ARM (and maybe the Rasp pi? he says with ulterior motives...)

[dscho]

Are there any plans to support an ARM64-compatible release of Git for Windows?

I looked into it, and the main issue is the lack of an ARM64 version of MSYS2. There does seem to exist an ARM64 version (at least a preliminary one) of Cygwin (on which MSYS2 is based), so there is hope.

However, I would like to encourage everybody who is interested in ARM64 support to chip in some work, as there is plenty to do before we will have a working Git for Windows on ARM64. Like, highly encourage.

I talk about MSYS2 because we need this for the parts of Git that are still implemented as Unix shell and Perl scripts (git bisect and git submodule are probably the two most prominent ones).

In the short run, if you should try to build Git for Windows yourself, in Visual Studio, by cloning this repository and checking out vs/master. Last time I checked, this failed to build the prerequisites, apparently vcpkg did not support building OpenSSL for ARM64 back then (this might have changed, see https://github.com/microsoft/vcpkg/releases for more details).

[ssimontis]

It's my understanding that you can only natively build ARM64 apps on Windows if they are based off of UWP. Otherwise, there is an x86 emulator for running Win32 applications, and Microsoft announced in November that they were working on an x64 emulator for Windows applications. I think we would need to wait until Microsoft makes x64 emulation generally available to approach this. If there's any preliminary work I can do in the mean time that would help out, please feel free to let me know and I will be happy to see what I can do.

[manbearian]

@dscho I'm interested in helping with bringing Git for Windows to ARM64. Locally i was able to get the vckpg dependencies built for ARM64 with only a tiny bit of hacking. The result of my efforts is a an ARM64 git.exe binary that at least dumps the help message successfully on my ARM laptop (i haven't tried anything else yet). Would you like to chat either here or offline about what you see as the full list of work that might be required beyond this?

ian Bearman
Principal Software Engineering Manager
Microsoft Visual C++ Team: Optimization & Code Generation
/* Making your code faster, smaller, smarter! */

[dscho]

That sounds amazing! Let's chat here, okay?

One of the biggest obstacles to getting Git to work on ARM is the lack of a Bash. But many things already work without a Bash...

So we could aim first for building a slightly incomplete Git for Windows (ARM64 edition) using the Visual C build.

What was that hackery you needed? Patches to vcpkg?

[manbearian]

I can create a PR for my changes, but i'm not sure the process for this. The changes are all in the 'git' source tree. Is there doc/contributors guide that covers this?

In the meantime, here's an outline of what i did based on SDK-64:

• project/sln changes (through perl script)
  • upgrade project to more recent VS (required to get ARM64 compilers, i picked latest, v142).
  • add ARM64 build configurations to projects/solution
  • added an item to the vcxprojs for VCPKGArch to be set correctly for ARM64
• source changes
  • update compat/bswap.h
• vcpkg changes
  • update compat/vcbuild/vcpg_install.bat to install both x64 and ARM64 packages
  • manually run the vcpkg_install script to install ARM64 packages

After these changes, building the git.sln for ARM64 "just works". I copied over to my ARM64 device and i have some very basic functionality working (i.e. the logo and help message). I dropped the binary into a folder layout matching an installed version and things got a bit better, but authentication for a fetch didn't work (missing DLLs i believe) and git checkout doesn't work if there are any new files (i didn't look further into this last week).

Is there a top-down view i can get of dependencies? That might be better than stabbing tin the dark looking for missing pieces. Additionally, i couldn't find documentation on building the installer, which i thought might give me a clue as to what binaries i need to look for.

[rimrul]

The documentation for building an installer is at https://github.com/git-for-windows/git/wiki/Making-an-installer

[manbearian]

Btw, i've tried to push my local changes to the git repo, but its not working for me...

C:\git-sdk-64\usr\src\git>"C:\Program Files\Git\cmd\git.exe" push --set-upstream origin initialArm64 remote: Permission to git-for-windows/git.git denied to manbearian. fatal: unable to access 'https://github.com/git-for-windows/git/': The requested URL returned error: 403
Ideas?

[rimrul]

You don't have push access to git-for-windows/git (very few people do). Click "Fork" near the top to create a copy on your github account, push there, then create a pull request.

[manbearian]

I'm making steady progress by mixing x86 binaries (from x86 install of Git for Windows) and the ARM64 binaries i built. I'm blocked on a few things right now and any help on these is appreciated:

1. trying to make progress on clone: i can now clone from a repo on github (e.g. my copy of git sources), however i cannot clone something in Azure DevOps. Likely related to authentication, but i'm seeing this message:

Cloning into 'c:\msvc1'...
fatal: Unsupported SSL backend 'openssl'. Supported SSL backends:
        schannel

i'm not sure where to start to investigate this, so pointers to what the authentication process or where its located in source at lest might give me a push in the right direction.

2. I successfully built a setup (thank for the instruction), but its not quite what i need for a few reasons: first the setup doesn't seem to contain the ARM64 git.exe that i built locally, not sure from where/what it grabbed its binaries. second, the setup didn't ask the usual Qs that setup asks, not sure if that is something i did wrong or not.

3. finally, related to setup: one path, which i think is a good one, in the direction toward getting something viable on ARM64 Windows is to get a setup that mixes 32-bit x86 and ARM64. (e.g. mingw32). This is because the emulator on ARM64 Windows only supports 32-bit x86 binaries.

Please let me know your thoughts on these topics.

thanks,
ian

[manbearian]

Spending time debugging fatal: Unsupported SSL backend 'openssl' has lead me to the conclusion that the libcurl i have is built without OPENSSL, (i see -DCMAKE_USE_OPENSSL=OFF in my build log) but that feature seems to be enabled in the version that i have installed. When i run git config --system -l i see http.sslbackend=openssl, and this setting seems to be sending git-remote-http.exe down the path that won't work.

The libcurl dependency is a vcpkg dependency... so i didn't run any configuration on it: is there possibly something strange here? Is this is a scenario that is not tested in VS build scenario?

Additional help is appreciated.

[rimrul]

Have you seen microsoft/vcpkg#9683?

To get openssl support with vcpkg curl you apparently need to run vcpkg install curl[core,openssl]

[manbearian]

@rimrul , thanks. that was my conclusion as well from reading the vckpg docs, but that still leaves open my second question: Is this is a scenario that is not tested in VS build scenario?

How far into uncharted territory am i right now?

[dscho]

How far into uncharted territory am i right now?

Far ;-)

Last time I tried to compile for ARM, vcpkg did not support building OpenSSL for ARM. But you might be able to build cURL with "winssl" (also known as "Secure Channel") support, which is provided by Windows, so there is no need to compile that yourself. This backend is essentially an alternative to using OpenSSL, and also allows cURL to connect via HTTPS.

[dscho]

Cloning into 'c:\msvc1'...
fatal: Unsupported SSL backend 'openssl'. Supported SSL backends:
schannel

Nevermind, I should have read back further: schannel is what you want to use. You can configure it via git config --global http.sslBackend schannel.

[manbearian]

Thank you for the workaround, @dscho , My 'uncharted territory' was more about the VS build then ARM64. From what i cnan teThe x86 and x64 vcpkg builds of curl also do not have openSSL support as it not included in the default build. Is the VS build intended to be fully functional?

[dscho]

Is there a top-down view i can get of dependencies?

Not really. I'll try to give you a very brief (and probably incomplete) overview:

Git for Windows consists of the parts that are implemented in C, and some Perl/shell scripts. The Perl/shell scripts are run via MSYS2's Perl and Bash. Unfortunately, Git assumes that a shell interpreter is available (yay for Linux legacy), which shows when you try to execute a pager (less.exe, which is also provided by MSYS2) or a hook, or try to override which program is used to connect via SSH.

The biggest problem with MSYS2 is that it is derived from Cygwin, and Cygwin started only recently looking into supporting ARM (I have not checked how far that effort went so far). Suffice to say, MSYS2 is not provided for ARM.

Apart from that, Git for Windows is actually independent from MSYS2. The parts that are implemented in C (which are nowadays almost all commands, notable exceptions are the submodule support, git bisect and git svn, those are still all scrips) are compiled into pure Win32 executables.

We usually use GCC to build those executables, in particular the mingw-w64 port of GCC.

We also provide a way to build with Visual C. Regarding this question:

Is the VS build intended to be fully functional?

We try our best to keep it as functional as possible, the CI/PR pipeline tries to verify that.

Note that the test suite itself is implemented in shell script, therefore you won't be able to run any of those regression tests if you don't have a shell interpreter (and we typically use MSYS2's Bash for that, more on that below).

So the "VS build" is actually a hodgepodge of executables compiled with Visual C, copied into the right place (I do not even think that we have an official MSBuild target for that), and Perl/shell scripts run via the parts from a MSYS2 system (which are compiled with yet another fork of GCC).

Now, for the dependencies. I will restrict myself to the C part here. The minimal set of dependencies consists of zlib. There is no way to run Git without zlib.

Git for Windows also uses these dependencies in addition:

• libintl for localization (although you can technically compile Git via the Makefile knob NO_GETTEXT),
• PCRE2 for regular expressions, Perl style (it is possible to compile Git without PCRE2, i.e. without support for git grep -P)
• libiconv to convert between encodings (again, this can be turned off with a Makefile knob: NO_ICONV)
• cURL (without cURL, you cannot fetch/push from/to HTTPS/HTTP URLs)

Now back to the shell scripts. There is BusyBox, and there is a Win32 port of it. BusyBox is an all-in-one executable that provides the functionality of a POSIX shell, of all the Unix utilities frequently expected by scripts, e.g. sed, awk, etc. We do have an experimental, non-interactive version of Git for Windows (the "MinGit edition" intended to be bundled and used by 3rd-party applications that do not want to depend on a full Git for Windows installation). The build process of BusyBox is derived from the Linux kernel's build system, i.e. it is pretty heavily tied into shell scripting and POSIX functionality. With a substantial amount of work, it might be possible to compile BusyBox-w32 for ARM. That would be our best bet, I think, to have a more complete Git for Windows on ARM.

For Perl, I do not really have any hope (and no, using Strawberry Perl is not an option: Git's heaviest Perl user is git svn and it requires the perl-svn bindings, i.e. native Perl modules to access Subversion's APIs, and I doubt that Strawberry brings those).

[dscho]

Is this is a scenario that is not tested in VS build scenario?

As you probably guessed: indeed, the regression test suite does not test HTTP/HTTPS functionality on Windows. The reason is that Git's regression tests covering that use case require Apache2 server executables to be available for starting up a transient HTTPS server.

[manbearian]

I was able to get curl building with openSSL and cloning from Azure DevOps worked!

Cloning into 'c:\msvc1'...
remote: Azure Repos
remote: Found 827484 objects to send. (9621 ms)
Receiving objects: 100% (827484/827484), 2.03 GiB | 12.29 MiB/s, done.
Resolving deltas: 100% (381713/381713), done.

BTW, this is about 2.5x faster clone than the x86 emulated Git for Windows 🏎 💨

[manbearian]

What do you think of my idea of building an ARM64 installer with x86 binaries? It seems like this could create a ramp to convert to native ARM64 step-by-step. Initially the package could contain ARM64 git compiled binaries and dependencies and then move to ARM64 mingw and other utilities once they are avaiable/supported?

[dscho]

Excellent news!

About the x86 binaries... Honestly, I'd like to avoid that, but it would make sense, wouldn't it? That way you have a fully functional Git for Windows, relatively fast in most commands...

[manbearian]

x86 emulation on Windows on ARM is quite good; my option is that developers shouldn't shy away from taking advantage of it.

[dscho]

Okay, that makes sense. So to build an ARM64 version of Git for Windows, we would start with a 32-bit Git for Windows SDK, build the ARM64 binaries via MSBuild, install them into the SDK and then bundle the result.

The installer itself is already an x86 executable (so that we can use the same framework for both 32-bit/64-bit).

The only tricky thing for now would be the step where the binaries are copied into the SDK; the command make install should do it, but at the moment it would insist on re-building everything with GCC.

[manbearian]

I'm hacking through the various pieces to put something together as proof-of-concept. After spending some time on it yesterday, i think some help understanding how the scripts and utilities in build-extra work might accelerate things. Is there any literature on the shell script methodology or installer tools that are used? Specifically i'm working on how the script is discovering what binaries to copy into the install package as i need to add some new ones, but generally in order to turn this into something more then just a hack, i think i need more details.

[manbearian]

Update: i've hacked up several files in the installation to build an appropriate installer and i have things basically working now from what i can tell. I ran my installer on my ARM64 Windows laptop and was able to clone a repo using the ARM64 git. The changes i've made to the installer are a mess though. Current state:

• Changes to the git Makefile
  • make install invokes msbuild (this can easily be parameterized)
  • make install installs more things (e.g. vcruntime140.dll and other libraries that have new names)
• Changes to release.sh script
  • the installer\release.sh script invokes the mingw installed git which won't work since its ARM64 now.
  • i have two workarounds depending on the situation:
    • hard code the results
    • invoke the version of git installed on the underly Windows OS.
  • Neither is ideal; Perhaps the idea of running make install is wrong and instead the script should grab binaries from another location?
• Changes to the pacman packages
  • i couldn't figure out a good way to add more file to the install package.
  • for now i've added the files i need to '/var/lib/pacman/local/mingw-w64-i686-git-2.25.0.1.7c71c859c9-1/files`
  • The added files that are new dependencies for MSVC built git.exe (e.g. vcruntime140.dll) or libs that changed names (e.g. libiconv.dll insted of libiconv-2.dll)

Pretty happy with where things are for a Friday afternoon!

[dscho]

@manbearian excellent progress!

FWIW I think the best way to address these things would be to hack something related to make-file-list.sh in build-extra: this is the script that determines what files to include in the installer/portable Git/MinGit.

There are already customizations in that script with regards to MinGit, and we can easily add more.

Having said that, my personal preference would be to build a mingw-w64-arm64-git. This package would, however, violate a couple rules: it would install into /mingw32/... even if ARM64 is everything but i686.

The way we build this package is via makepkg-mingw in /usr/src/MINGW-packages/mingw-w64-git (which is tracked here).

To build the -arm64- package, I would suggest adding an optional package (much like we allow skipping the -pdb package). It would probably make most sense to piggy-back that on top of the i686 build (i.e. make it dependent on an environment variable like TARGET_ARM64 and on CARCH). When building the -arm64- package, we should probable not build any other package, i.e. override pkgname.

Finally, it would be really awesome if we could build the binaries via make MSVC=1 (with a further variable to specify to target the ARM64 architecture, of course). That would make it the easiest to build this package, as we would not require all kinds of dances to figure out where MSBuild happens to be on the current machine.

[dscho]

Now the PR is merged, can we expect it an ARM64 binary will be released with next version/build?

@tommyvct I have to admit that the tone of this almost-demand rubs me in all the wrong ways. You may not have meant to sound like it, but it sure comes over as if you considered this a thing other people do for you and all you have to do exactly nothing but sit back. Comments like this make me want to stop spending any time on that project. It is very demotivating.

[tommyvct]

Now the PR is merged, can we expect it an ARM64 binary will be released with next version/build?

@tommyvct I have to admit that the tone of this almost-demand rubs me in all the wrong ways. You may not have meant to sound like it, but it sure comes over as if you considered this a thing other people do for you and all you have to do exactly nothing but sit back. Comments like this make me want to stop spending any time on that project. It is very demotivating.

I'm sorry if my tone was not appropriate.

You see that this issue has been inactive for 5 months, and I just want to have an update on the situation as an outsider, and potentially what I can do.

I feel sorry that your comments also sounds demotivating. You may not in a good mood, but Christmas is fast approaching, why not take a break?

Merry Christmas 🎄

[Alovchin91]

@dscho While I totally understand and also feel that the coronavirus situation is getting on all of us and hence there is a lot of frustration in the air, I'm pretty sure @tommyvct didn't mean anything wrong and is probably just not a native English speaker. Me too, btw, so please keep this in mind 🙂

In any case, I believe we are all very supportive of you and this project. I personally am fully open to support your efforts as much as I can, whether by providing my Surface Pro X as a build/test ground, picking up some tasks (please let me know what exactly needs to be done there), or donating to the project.

Please stay safe and try to still have fun in these dark times! We're all in the same boat ✋

[dscho]

@Alovchin91 thank you for providing a bit of motivation again. Much appreciated.

[walbourn]

I'd very much appreciate an ARM64 native build of the git tools when working on my Surface Pro X. The emulation using the x86 binaries works for basic usage, but it fails with high-demand scenarios. For example, git lfs filtering fails when trying to fetch a 1 GB binary...

git clone https://github.com/walbourn/directxtexmedia.git

[Hyncha]

Can't wait for git to run natively on windows 10 on ARM, would love to help but I really have no clue :(

[tommyvct]

@walbourn That's indeed a git-lfs bug. You may wanna create an issue over there.
Issue created. Don't forget to subscribe.

Meanwhile you can try build the thing to arm64 to see if that helps.
I tried. The official go compiler cannot make windows arm64 binaries.
I tried again. The official go compiler CAN make windows arm binaries, but it still crashes for your case.
I tried 386 binary in a virtual machine and it still crashes. Strangely x64 binary didn't.

[dennisameling]

Let me try to summarize our progress here. The main goal currently is to create an arm64 package that has the mingw32 binaries in it (32-bit emulation, things like bash/curl/etc.), which significantly speeds up our progress for now. The only exception to this is git.exe (and related binaries) itself, which can be compiled for arm64 natively now. This already shows a ~2x speed improvement over the emulated 32-bit version 🚀

Steps to get to a first arm64 release (again, mostly with mingw32 binaries) - please correct me if I'm missing something:

• ✔️ Enable compiling Git for Windows (git.exe etc.) natively for arm64 Windows arm64 support #2915 (comment)
• ✔️ Fix double .exe suffix in CMake generated build Fix double .exe suffixes in CMakeLists.txt  #2966
• ✔️ Include vcpkg DLLs in CMake build cmake(install): include vcpkg dlls #2971
• ✔️ Add ARM64 to /etc/profile added MSYSTEM=ARM64 case in /etc/profile git-sdk-32#6
• ✔️ Update /etc/profile in git-extra.install.in Start support for MSYSTEM=ARM64 build-extra#321
• ✔️ Modify git-wrapper in mingw32 to use the native git.exe, set MSYSTEM=ARM64, etc. Add arm64 support to git-wrapper MINGW-packages#44
• ✔️ Create portable version of ARM64 Git for Windows portable: add arm64 artifact option build-extra#323
• ✔️ Create MinGit package for ARM64 Git for Windows mingit: add arm64 artifact option build-extra#324
• ✔️ Create archive package for ARM64 Git for Windows archive: add arm64 artifact option build-extra#325
• ⏳ Create an installer for ARM64 installer: add arm64 artifact option build-extra#326
• ⏳ Add ARM64 artifacts to CI

Here's a rough first arm64 version for anyone who wants to try: https://github.com/dennisameling/git/releases/tag/2.30.0-beta2

[dscho]

• Add ARM64 to /etc/profile git-for-windows/git-sdk-32#6 and git-for-windows/git-sdk-64#32

Sounds good, we only have to do that in the post-install part of the git-extra package defined in build-extra, as I mentioned here.

• Modify git-wrapper in mingw32 to use the native git.exe, set MSYSTEM=ARM64, etc. git-for-windows/MINGW-packages#44

Right. I still hope that you can get an i686 version of git-wrapper.exe to determine whether it is really running in Windows/ARM64.

[rimrul]

That should be possible to determine using IsWow64Process2().

[dscho]

That should be possible to determine using IsWow64Process2().

Heh, I just added this suggestion 15 minutes ago: git-for-windows/MINGW-packages#44 (comment)

[rimrul]

According to this blog post from 2017 it should work, but we might need to be careful on Windows 10 prior to 1709.
We might be able to get by with just using it via compat/win32/lazyload.h which we should do either way to avoid breaking the x86 build for Windows versions below 10.

[dennisameling]

Good idea, thanks for sharing! Will check it now

[dennisameling]

It works with IsWow64Process2()!! 🎉 Just tested with my simple demo C++ project.

x86/i686 executable on ARM64 device:

x86/i686 executable on x64 device:

I'll spend some time this weekend to see how I can incorporate this in the git-wrapper.c together with lazy loading, to prevent issues with Windows versions lower than Windows 10 1709. To be continued 🚀

[dscho]

According to this blog post from 2017 it should work, but we might need to be careful on Windows 10 prior to 1709.
We might be able to get by with just using it via compat/win32/lazyload.h which we should do either way to avoid breaking the x86 build for Windows versions below 10.

Well, we're talking about the Git wrapper here, so it would need to do its own LoadLibraryExW() thing instead of using the niceties of lazyload.h.

[rimrul]

git-wrapper.c is part of mingw-w64-git and the PKGBUILD file lists git-for-windows/git as part of its sources, so shouldn't we be able to just do #include "compat/win32/lazyload.h" in git-wrapper.c?

[dennisameling]

Got it to work (including on Windows 8.1, haven't tested Win7) in git-for-windows/MINGW-packages#44 🎉

[dscho]

git-wrapper.c is part of mingw-w64-git and the PKGBUILD file lists git-for-windows/git as part of its sources, so shouldn't we be able to just do #include "compat/win32/lazyload.h" in git-wrapper.c?

So far, git-wrapper.c does not include anything from Git's source code, to avoid having to link libgit.a.

I suppose that lazyload.h does not require libgit.a to be linked in, but it sure muddies the water if we allow that header to be #included but no other of Git's header files.

[rimrul]

I would've liked the internal consistency of doing this kind of dynamic function loading the same way throughout the Git for Windows code base, but I see your point.

Seems like Dennis already picked the loading implementation from the blog post anyways.

[dennisameling]

Update: we're making great progress. I was able to create a PortableGit installer for ARM64, you can try it out here: https://github.com/dennisameling/git/releases/tag/2.30.0-beta2 🚀

To see an overview of open PRs for the ARM64 work please see this comment.

Here's how it works:

This is basically a mingw32 build, but with some logic to recognize if the user is running on an ARM64 device. If they are, it will look first in an additional directory called arm64, which has some native binaries in it like git.exe. This is what will give you the ~2x speed boost in commands like git clone. Other binaries like curl aren't available natively for ARM64 yet, so Git Bash will fallback to mingw32 for those binaries.

So it first looks in /arm64 (1) and if the binary is not available there, falls back to /mingw32 (2):

Here you can see that it indeed uses Git from /arm64/bin/git:

We're getting very close everyone! Exciting times 🎉

[0cyn]

@dennisameling seems like that release was removed, any word on it?

[dennisameling]

@KritantaDev sorry, I had to re-fork this repo. Just uploaded the beta release again on the same URL, could you please try again now? 😊