[WIP] Arm64/Aarch64 ci builds #4862
Comments
|
i'm somewhat close after two solid days of head scratching. i ran a built debug version on a RPi3 and got an error relating to V8. |
|
Feeling like I've hit somewhat of a dead end myself, and don't know enough about rust and the build process to do this justice. Passing this on to whoever wants to work on this. |
|
|
|
if it helps, |
|
(for my reference, first round of attempts: sunsetkookaburra/deno@2ef273a) |
|
If you can get v8 to compile successfully the only other major issue you are going to run into is snapshots. Snapshots generated using a x86 compiles of v8 are not compatible with arm compiles of v8. It took me a long time to figure this out the first time around, but It makes more sense when you realize that a snapshot is mostly compiled bytecode from v8's JIT compiler. The only solution I managed to get working here is just don't use snapshots for arm builds, but I would be happier if we could generate arm snapshots. |
|
I did try (at one point) forcing it to build v8 (on arm), and that didnt work. |
|
How can i reproduce that kind of behavior on my setup? Or this just can be reproduced on the CI? |
|
@ktfth this was trying to build using the ci, but following the steps for aarch64 on your machine should do much the same. Rust-Cross looks promising, but I couldn't figure out how to install python in that image or rust, it also needs docker installed locally. Rust cross (https://github.com/rust-embedded/cross) is so close to an answer I feel, you just need to figure out installing the build tools onto that image (rust+python+...) |
|
Running the following command "cross build --target aarch64-unknown-linux-gnu" I have getted the output: error: failed to run custom build command for Caused by: --- stderr warning: build failed, waiting for other jobs to finish... You have passed this step? |
|
@ktfth I did manage to. I used I used this dockerfile with rust-cross: (I didn't try using ADD) |
|
It's a good place to put docker file on the project? https://github.com/ktfth/deno/blob/feature/aarch64/Dockerfile |
|
I have updated the docker file to act as multi-stage and work well now i need to solve that part. error: failed to parse manifest at Caused by: Caused by: this Cargo does not support nightly features, but if you Have any tip for that? |
|
The build with "cross build --target aarch64-unknown-linux-gnu" work well |
|
Have you managed to run the binary successfully? I did manage to build it, but at runtime it would fail. |
|
note to self, i found this in the v8-flags |
|
@sunsetkookaburra So I have a Raspberry Pi 4 (8GB) that's running the 64bit Version of Raspberry Pi OS. And I have another version of Raspberry Pi 4 (4GB) that's running the 32bit version of Raspberry Pi OS. What can I do to help you? Do you have a list of requirement that are needed to be installed so that people like me can run along with you? I'm a C programmer, but only in so far as I've written some C programs. I'm not actually good at it. But I'd love to get Deno working on the Raspberry Pi as that's the vast majority of my development target. |
|
@Dygear First step is to follow the general instructions for installing deno for building it, if your doing it locally. To run it on my RPi3, I use a function of github actions to bundle a file from the build. If you look at past issues about Arm64/Aarch64 there was something about snapshots that I wasn't able to figure. I did try for a while (buried somewhere in those commits). There is a Best help I can give is just to look through the commits that I did make in that fork, and maybe try them either on your RPi or on your own fork. And if they're still there, to look at the github actions and try and find ones from my pull-request... https://github.com/sunsetkookaburra/deno/actions?query=workflow%3Aci I do feel like I got close, so maybe do some digging around the V8 part of the build - there is some stuff to do with snapshots being arch/os specific, from what I recall (to do with compilation). |
|
@sunsetkookaburra Do we have somewhere, where we can have an on going conversation about this? I'm on the Deno Discord server with the username |
|
Having gotten this working in the past I figure I should help clear up some of the confusion here. You can already build and run deno directly on most arm64 platforms(as long as you don't run out of memory compiling). Platforms I have already tested include a pi 4 4gb, pi 3 1gb, and several Android phones(and tablets) using termux proot and Ubuntu. This should be pretty simple on most distros. Just clone the main deno repo and submodules and run 'cargo build'. You might need to install a few deps on most distros. On Ubuntu I only needed to install gcc, rust, and git, but this might very a bit on your install. For armhf/armv7/arm32 platforms things are a little more complicated, since we don't currently have a build process for 'rusty_v8'. I started poking at this part today, but I think it's going to be a lot more complicated than just copying and modifying the arm64 build. For most people I would just avoid 32-bit arm platforms and os distros if possible. The cross compile situation is very complicated. It all revolves around snapshots. When v8 loads Javascript code into memory it essentially compiles it to bytecode(bit of a simplification here) hence v8 being a JIT compiler. On a x86 platform running a native executable this means that the bytecode is x86 bytecode. When we create a snapshot of v8 state it mostly includes this bytecode. If we are cross compiling this snapshot is generated by running a native executable ('build.rs') thus the snapshot generated is native to the compiler host and is not compatible with the platform we are compiling for. If we can either find a way to execute a arm native version of the snapshot generator at compile time or disable snapshots for cross compiles(my previous solution), we should be able to cross compile for arm64. I hope this clears up where arm support is at currently and what issues still need to be solved here. |
|
@afinch7 Thanks! That was helpful. I blew away my older install for Raspberry Pi OS 64 Bit and started fresh. For those that want to follow along, I'm using a Raspberry Pi 4 8GB with a 64 GB SD Card (Samsung Pro Endurance), the official power supply and the FLIRC case. I downloaded the latest version of Raspberry Pi OS Buster arm64 from here. I'm using the 2020-08-20-raspios-buster-arm64.zip file. I copied that over using the Raspberry Pi Imager. Once booted I enabled SSH and did the rest of the setup from an SSH session. I did the normal.
During the failed build process I noticed this ... Good, it sees that I'm an arm64 system. I noticed that @afinch7 said just run
Full build log can be found here for both the failure and success. I re-read your last paragraph @afinch7 and I realize the problem you are facing is much larger because of CI builds. So you are attempting to solve the systematic problem of CI on cross platform code. I think what you really need is a Raspberry Pi 8GB in order to check the CI build on that platform. The cross compile JIT giving you x86 byte code instead of ARM native byte code seems like a total nightmare. Perhaps it would simply be better if the arm builds happened on a native arm stack ... On a fairly cheap Raspberry Pi 4. Just for reference a fresh / cold |
|
After install the nightly version of rustup for Looks like a dependency problem with |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
@Dygear I would much, much prefer to just get a copy from a trusted source instead of compiling my own. Do you have a link to where I can find that? |
|
@tychi-nflx I provide builds over at https://deno.toaxl.com |
|
@crowlKats you're the best! thank you so much! |
|
@crowlKats Tested on arm64 linux-5.15 debian on a RPI4 4GB and works perfectly, all I had to do was to update libc. Thanks a lot! |
|
Are these custom builds really still needed? I just successfully installed deno on Android (aarch64) using Termux and proot (Debian) using just the standard (Don't get me wrong I would still love to see a native Android build that doesn't require proot but that's not what those builds people here are providing are for) |
|
No clue where that build is getting pulled from, https://packages.debian.org/search?keywords=deno&searchon=names&suite=all§ion=all |
|
Oh, whoops that's my bad, indeed that build is not from apt/pkg at all. I had forgotten I had dragged one of those custom builds into Just for completeness, if anyone is interested, here's the Android installation steps I used for proper reproduction this time: (aswell as putting deno on PRoot's # If you already have a PRoot distro set up you can just login to it and skip to the 3rd command.
apt update && apt install proot-distro -y
proot-distro install debian && proot-distro login debian
wget https://github.com/LukeChannings/deno-arm64/releases/download/v1.19.0/deno-linux-arm64.zip && unzip deno-linux-arm64.zip && mv deno /usr/bin/deno && rm -f deno-linux-arm64.zip
deno --version |
This PR supercedes #1137 as the rebase was much too gnarly. This is essentially that same PR. Copying the description w/updated status below: We should not merge this PR and release ARM support until we can run the latest `rover supergraph compose` and `rover dev` commands on ARM. The following must happen, in order, before landing this PR. - [x] Test ARM support with `federation-rs` release candidates - [x] Release[`federation@2.0.6-rc.0` _(javascript)_](https://github.com/apollographql/federation) - [x] Release[`supergraph@2.0.6-rc.0` _(rust)_](https://github.com/apollographql/federation-rs) - [x] Test downloading and running `supergraph compose` with `2.0.6-rc.0` in a `supergraph.yaml` with `rover` built from `main` natively on an M1, and in Docker on an M1 - [x] Continue releasing release candidates until it all works. ~Hopefully it is just `2.0.6-rc.0`. If that works, it is safe to assume 0.x will work, so no release candidate will be needed there.~ _update: it was not that easy, the next two steps are new_ - [x] Remove the CI builds for `aarch64-apple-darwin` apollographql/federation-rs because [V8 snapshots cannot be cross-compiled from x86_64 machines](denoland/deno#4862 (comment)) and [CircleCI does not currently provide native ARM64 Mac machines](https://circleci.com/docs/2.0/using-arm/#m1-mac-support) meaning we _have_ to cross-compile if we want ARM binaries at all. If either of those things change, we will revisit this decision. - [x] Test downloading and running `supergraph compose` with `2.0.6-rc.1` in a `supergraph.yaml` with `rover` built from `main` natively on an arm ubuntu device, and in Docker on an M1 - [x] Release first stable `federation-rs` releases with native ARM support (only Linux binaries, no ARM64 MacOS binaries) - [x] Release [`federation@2.1.0` _(javascript)_](https://github.com/apollographql/federation) - [x] Release [`supergraph@2.1.0` _(rust)_](https://github.com/apollographql/federation-rs) - [x] Release [`federation@0.37.0` _(javascript)_](https://github.com/apollographql/federation) - [x] Release [`supergraph@0.37.0` _(rust)_](https://github.com/apollographql/federation-rs) - [x] Test downloading and running `supergraph compose` with the packages we just released - [x] Release [`router@1.1.0`](https://github.com/apollographql/router) - [x] Prepare ARM support for Rover - [x] Cut `rover@v0.9.2-rc.0` - [x] Test downloading and running `supergraph compose` natively on an M1, and in Docker on an M1 - [x] Release more rcs until we are confident everything works - [x] Wait for the cloud router to default to version 2.1.x
|
I know it isn't super critical right now but any progress on this? |
|
Not super critical but definitely a good to have in Deno 🤩 |
|
I tried the following instructions #4862 (comment) with deno 1.35.0 but I get an error when starting deno no such file or directory (the file does exist) it seems more like a proot issue |
|
Seems like I have a 32bit arm android ( |
|
Why did this stall? |
This implements officially blessed and tested deno binaries for ARM64. Thanks to @LukeChannings for his tireless work in maintaining the deno-arm64 [1] repo, without which this project would have been far more complicated. For those of you requiring support for older GLIBC versions, that repo may still be required for the near future. Limitations: - This initial build is built on Ubuntu 22 using the stock GLIBC, which will limit the utility of these binaries in certain use-cases (eg: early versions of Ubuntu). We will attempt to support earlier versions of ARM64 GLIBC in a later revision. - Like the stock Linux x64 build, this is not a static build and requires GLIBC. Running on Alpine will require installation of GLIBC. Fixes #1846, #4862 [1] https://github.com/LukeChannings/deno-arm64
|
Completed by #22298 |
This implements officially blessed and tested deno binaries for ARM64. Thanks to @LukeChannings for his tireless work in maintaining the deno-arm64 [1] repo, without which this project would have been far more complicated. For those of you requiring support for older GLIBC versions, that repo may still be required for the near future. Limitations: - This initial build is built on Ubuntu 22 using the stock GLIBC, which will limit the utility of these binaries in certain use-cases (eg: early versions of Ubuntu). We will attempt to support earlier versions of ARM64 GLIBC in a later revision. - Like the stock Linux x64 build, this is not a static build and requires GLIBC. Running on Alpine will require installation of GLIBC. Fixes #1846, #4862 [1] https://github.com/LukeChannings/deno-arm64
Will create a PR when I have this working and cleaned up, at the moment the repo-history is a mess. Utilising some of the work from rusty_v8's arm 64 ci.yml done by afinch7. I'm trying out a community gh-action that allows running on non-intel cpus by utilising docker. I'm also looking into cargo runners, which I think provide the list of executables which can be stored and run on this docker instance. 🤞
The text was updated successfully, but these errors were encountered: