Platform requirements for Node.js 18

[richardlau]

Node.js 18 is due out in April 2022 and is planned to last until April 2025 (https://github.com/nodejs/Release/blob/main/schedule.json). We need to plan out any required toolchain/hardware changes we want/need to make.

cc @nodejs/build

What we build and test on

From https://github.com/nodejs/node/blob/master/BUILDING.md#official-binary-platforms-and-toolchains

Binary package	Platform and Toolchain	Decision	done
aix-ppc64	AIX 7.2 TL04 on PPC64BE with GCC 8	move minimum hardware to Power 8
darwin-x64	macOS 10.15, Xcode Command Line Tools 11 with -mmacosx-version-min=10.13
darwin-arm64 (and .pkg)	macOS 11 (arm64), Xcode Command Line Tools 12 with -mmacosx-version-min=10.13
linux-arm64	CentOS 7 with devtoolset-8 / GCC 8	Need to migrate (#2741)
linux-armv7l	Cross-compiled on Ubuntu 18.04 x64 with custom GCC toolchain
linux-ppc64le	CentOS 7 with devtoolset-8 / GCC 8	Need to migrate (#2741)	✔️
linux-s390x	RHEL 7 with devtoolset-8 / GCC 8	Need to migrate (#2741)	✔️
linux-x64	CentOS 7 with devtoolset-8 / GCC 8	Need to migrate (#2741)	✔️
win-x64 and win-x86	Windows 2012 R2 (x64) with Visual Studio 2019

Compiler levels

Operating System	Compiler Versions
Linux	GCC >= 8.3
Windows	Visual Studio >= 2019 with the Windows 10 SDK on a 64-bit host
macOS	Xcode >= 11 (Apple LLVM >= 11)

Actions

• Check with IBM if remaining on Power 8 until 2025 is viable (we'll need additional machines if not).
• Any reason to move the minimum deployment targets for macOS on from macOS 10.13?
• CentOS 7 gets updates until 30 June 2024 which is within the expected lifetime of Node.js 18, so we'll need to move (probably to RHEL 8, Consider moving from Centos7 to RHEL8 for 18.x #2741).
  Refs: https://wiki.centos.org/FAQ/General#What_is_the_support_.27.27end_of_life.27.27_for_each_CentOS_release.3F
• Do we need to add Windows 11 somewhere to the CI? @joaocgreis

[rvagg]

Any reason to move the minimum deployment targets for macOS on from macOS 10.13?

No really good reason, but I'm wondering if we should just adopt a practice of incrementing this by 1 at this time every year. Then we don't have to do big jumps like we've done a couple of times. Being a few major versions behind seems reasonable enough I think and I don't recall any complaints (ever) from users about macOS compatibility—in fact the only complaint I recall was a Node core dev who insisted on running a couple of versions behind the latest macOS, but that was within the value we set for this.

RHEL 8

What are our IaaS options for this? Do our providers have RHEL options, or can we switch a CentOS machine into RHEL mode somehow? Or should we be considering containierising this build?

[targos]

It might be good to use a higher Windows server version even if it's not based on Windows 11. 2012 R2 is quite old.

[richardlau]

RHEL 8

What are our IaaS options for this? Do our providers have RHEL options, or can we switch a CentOS machine into RHEL mode somehow? Or should we be considering containierising this build?

Not ideal. Unsurprisingly IBM Cloud offer RHEL options but neither Digital Ocean nor Rackspace currently do (#2741 (comment)) although they do provide Alma/Rocky. Red Hat do provide a tool for switching (https://www.redhat.com/en/blog/introduction-convert2rhel-now-officially-supported-convert-rhel-systems-rhel) which I've not tested yet. Containerization or custom VM images are other things on my list of things to investigate.

I have an action to obtain a RHEL subscription for the Node.js project (possibly via https://www.redhat.com/en/blog/extending-no-cost-red-hat-enterprise-linux-open-source-organizations) and see if we can make it work for us (i.e. the build wg). Having a subscription would immediately simplify set up for the existing RHEL7 LinuxOne machines (at the moment we have to manually copy rpms over to set up the compiler toolchain).

[AshCripps]

No really good reason, but I'm wondering if we should just adopt a practice of incrementing this by 1 at this time every year.

+1 on this as apple can be quite aggressive on dropping support for its OSes.

[sxa]

Particularly with the new Ampere Altra systems being available, we should also consider whether we need to continue to cross-compile for arm32 or whether to use an arm32 docker container on one of the Altrea aarch64 hosts for building to reduce the complexity and increase commonality with how other platforms are built.

[mhdawson]

To add to @richardlau's comment above our providers that don't have RHEL8 images built in do allow you to bring your own image(I think that is what @richardlau meant by customer VM images). I think that should be relatively easy to do and we should be able to get the images under the program to let Open Source projects use RHEL.

[mhdawson]

@joaocgreis thoughts on the Windows levels?

[rvagg]

From the trusty https://en.wikipedia.org/wiki/Glibc#Version_history table, switching to RHEL8 would give us a minimum glibc of 2.28 which ships with Ubuntu 18.10, Debian 10 (Buster), and Fedora 29. The only concern there is that it's later than 18.04, so we're ruling out those systems. But I think that's reasonable for a new release of Node.js - 18.04 users will either need to stick to older Node.js or compile themselves using a newer toolchain from ppa.

[rvagg]

Re ARMv7, our cross-compiler-ubuntu1804-armv7-gcc-8 label currently uses an 8.3.0 compiler (see), and it's on an 18.04 base so pretty compatible. As long as the 8.3.0 minimum is solid then that should be good to go I think.

[targos]

Here are some end of support dates for Windows and Visual Studio. Found on https://docs.microsoft.com/en-us/lifecycle/products/

I did not count "Extended Security Update Year X", as the documentation says:

The Extended Security Update (ESU) program is a last resort option for customers who need to run certain legacy Microsoft products past the end of support.

• Windows 8.1: Jan 9, 2023
• Windows 10 Enterprise and Education: Oct 13, 2025
• Windows 10 Home and Pro: Oct 13, 2025
• Windows Server 2012 R2: Oct 9, 2023
• Windows Server 2016: Jan 11, 2027
• Windows Server 2019: Jan 8, 2029
• Visual Studio 2015: Oct 13, 2025
• Visual Studio 2017: Apr 12, 2027
• Visual Studio 2019: Apr 9, 2029

[targos]

Here's my suggestion based on ^

nodejs/node#42105

[targos]

We found out in nodejs/node-v8#220 that V8 now uses a kernel feature that is only available in Linux 3.17 (glibc 2.27).

[rvagg]

@targos have we committed to that version of V8 for Node 18?

If so, that messes with our Linux strategy quite a bit, we'll have to bump minimum build requirements to:

• Ubuntu 18+
• RHEL / CentOS 8+
• Debian 10+

Maybe devtoolset gets us out of some trouble for runtime requirements for our shipped binaries, but I suppose we need to test that they're compiling—do we have any runs of this through the release pipeline yet for v8-canary builds, or are we still at the integration phase?

[richardlau]

@targos have we committed to that version of V8 for Node 18?

If so, that messes with our Linux strategy quite a bit, we'll have to bump minimum build requirements to:

* Ubuntu 18+

* RHEL / CentOS 8+

* Debian 10+

Maybe devtoolset gets us out of some trouble for runtime requirements for our shipped binaries, but I suppose we need to test that they're compiling—do we have any runs of this through the release pipeline yet for v8-canary builds, or are we still at the integration phase?

For that particular V8: https://ci.nodejs.org/job/node-test-commit/52045/
Basically any CentOS 7/RHEL 7 Linux failed but the other Linux distributions built/tested successfully. Ubuntu 18.04 uses glibc 2.27 (where the API being used by V8 was introduced) and passed.

I'm planning to get some RHEL8 x64 instances set up for the CI this coming week to get some test binaries to try out on the various other distributions. I'm fairly certain though that we'll need at least glibc 2.28 to run the binaries, which as noted by you before (#2815 (comment)) would look to rule out Ubuntu 18.04. AFAIK we don't have glibc checks at build time though, so anyone on those could still attempt to build from source.

We're going to have to move off CentOS 7 for Node.js 18 unless we want to be running on an unsupported OS at the tail end of Node.js 18's expected lifespan, so a glibc bump is going to happen (CentOS 7/RHEL 7 are glibc 2.17).

[targos]

have we committed to that version of V8 for Node 18?

We have always tried to include the most recent possible V8 version available on release day.

do we have any runs of this through the release pipeline yet for v8-canary builds

Latest canary release build has this: https://ci-release.nodejs.org/job/iojs+release/8275/
Integration is done with GitHub actions, which are able to build it.

[mhdawson]

@rvagg I think we were already planning on moving up to RHEL 8 for 18.x due to the EOL for REHL 7.

[richardlau]

Re ARMv7, our cross-compiler-ubuntu1804-armv7-gcc-8 label currently uses an 8.3.0 compiler (see), and it's on an 18.04 base so pretty compatible. As long as the 8.3.0 minimum is solid then that should be good to go I think.

After April 2023 Ubuntu 18.04 requires an Ubuntu Advantage subscription to get further updates, which is two years before Node.js 18 goes End-of-Life: https://ubuntu.com/about/release-cycle
Do we want to move to something else? April 2023 is also a year before Node.js 16 goes End-of-Life, so we're potentially exposed there.

[richardlau]

Re ARMv7, our cross-compiler-ubuntu1804-armv7-gcc-8 label currently uses an 8.3.0 compiler (see), and it's on an 18.04 base so pretty compatible. As long as the 8.3.0 minimum is solid then that should be good to go I think.

After April 2023 Ubuntu 18.04 requires an Ubuntu Advantage subscription to get further updates, which is two years before Node.js 18 goes End-of-Life: https://ubuntu.com/about/release-cycle Do we want to move to something else? April 2023 is also a year before Node.js 16 goes End-of-Life, so we're potentially exposed there.

Speaking with @sxa it sounds like it might not matter so much which OS is being used and it's more the cross compiler, with the GLIBC version being linked against coming from the cross compiler in https://github.com/rvagg/rpi-newer-crosstools.

[richardlau]

Having said that, https://ci.nodejs.org/job/node-cross-compile/37193/nodes=cross-compiler-ubuntu1804-armv7-gcc-8/ failed for #2815 (comment) so our cross compiler is linking against a glibc older than 2.27. Although the particular commit was patched out (nodejs/node@ec77f2d) I have no idea if we'll just get another down the line. V8 use clang themselves so I don't think they have a notion of minimum glibc support.

[rvagg]

btw, I do have a commitment from Canonical to sponsor us with some ESM licenses for our infra, I've just failed to follow up on that due to being very occupied with other things! it was mainly interesting for some of our infra that's a bit stuck on older versions and are painful to upgrade but we can use it in our testing infra too.

[targos]

V8 just landed v8/v8@4e81f25, which reverts the glibc bump requirement on that side.

[targos]

Since it's a runtime check, we would still have to compile using a newer version of glibc:

https://ci.nodejs.org/job/node-cross-compile/37428/nodes=cross-compiler-ubuntu1804-armv7-gcc-8/consoleFull

Is it going to happen before the release of v18.0.0 or I should reintroduce the workaround?

[richardlau]

I don't know what we're going to do about the cross-compiler for armv7 -- the glibc compiled against is the one in the cross compiler and not the host (ubuntu 18.04 in this case).
arm64 will be moving (it's dependent on getting another release machine from Equinix which is in progress).

[richardlau]

I don't know what we're going to do about the cross-compiler for armv7 -- the glibc compiled against is the one in the cross compiler and not the host (ubuntu 18.04 in this case).

I had a chat with @sxa and he suggested we could try building armv7 in an armv7 container on the arm64 machine(s) and avoid cross-compilation altogether.

[rvagg]

Not cross-compiling is certainly an option, if you want to pursue that then maybe that'll be a decent solution. But, because it's possible, I went ahead and made a copy of the gcc 8.3.0 cross-compiler but with glibc 2.27 rather than 2.24. I don't have time to try it out right now but if someone does, it's here on a branch: rvagg/rpi-newer-crosstools#1

Unfortunately it's a bit involved to wire it up -- if it's still working like last time I touched it, something like:

• Edit ansible/roles/docker/files/cc-selector.sh to add it as a new option
• Add a new label to the cross-compiling nodes in Jenkins (ci and ci-release both use labels for selection)
• Edit jenkins/scripts/VersionSelectorScript.groovy to shunt the right Node.js version to the new label
• (edited): Re-deploy the cc-selector.sh and the rpi-newer-crosstools onto the cross compiling machines -- since I think we do this in Docker contains in ci (don't recall ci-release, maybe the same), this might be fiddly, and maybe it'd be easier if I merged that PR so it's master, let me know.

The node running the build extracts the label that was chosen ($label) and matches that to the right compiler, this is all done in cc-selector.sh.

We probably shouldn't touch existing 8.3.0 use, bumping glibc would be a breaking change for some users.

[richardlau]

@rvagg I tried rvagg/rpi-newer-crosstools#1 locally in a container and it appears to work. I'm happy to do the wiring on the deploy to the cross-compiling machines/Jenkins side.

What do you think about bumping to glibc 2.28 to be consistent with all of the other Linux archs that we are building on RHEL 8? FWIW I put together a Linux distribution->glibc map -- while glibc 2.28 would rule out Ubuntu 18.04, Debian 10 is glibc 2.28: https://gist.github.com/richardlau/6a01d7829cc33ddab35269dacc127680

Also FWIW the gcc 8.3 cross compilation toolchain from https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-a/downloads is glibc 2.28.

We probably shouldn't touch existing 8.3.0 use, bumping glibc would be a breaking change for some users.

100% agree.

[rvagg]

@richardlau yep, pushed a new version that uses 2.28. crosstools-ng is a bit out of date but it does at least have a good range of glibc that we are about that it'll deal with without too much hassle.

[richardlau]

@rvagg I had a go at the cc-selector/Jenkins changes required: #2912

The cross compilation looks like it's working but unfortunately I'd forgotten that although the Pi's are running Debian 10 the tests are being run in a Debian 9 container which only has glibc 2.24. I think I've made the required Ansible changes to add a Debian Buster container, but ran into trouble trying to run the playbook against the Pi's (the first one I picked was one of the offline ones and the second one failed at the package upgrade step).

[targos]

nodejs/node#42659 just landed. Are we done?

[richardlau]

nodejs/node#42659 just landed. Are we done?

The only thing that hasn't been done is updating the Pi's (#2914 (comment)) but I think we can release Node.js 18 without them (we have some armv7l testing run in containers) -- I have temporarily stopped https://ci.nodejs.org/job/node-test-binary-arm-12+/ running for Node.js 18 and later so the CI will pass (#2912 (comment)).