install using "optionalDependencies"

[evanw]

This PR changes esbuild's installation strategy in an attempt to improve compatibility with edge cases such as custom registries, custom proxies, offline installations, read-only file systems, or when post-install scripts are disabled.

The old installation strategy manually downloaded the correct binary executable in a post-install script. The binary executable is hosted in a separate platform-specific npm package such as esbuild-darwin-64. The install script first attempted to download the package via the npm command in case npm had custom network settings configured. If that didn't work, the install script attempted to download the package from https://registry.npmjs.org/ before giving up. This was problematic for many reasons including:

• Not all of npm's settings can be forwarded due to npm bugs such as [BUG] Config environment variable forwarding breaks with false config values npm/cli#2284, and npm has said these bugs will never be fixed.
• Some people have configured their network environments such that downloading from https://registry.npmjs.org/ will hang instead of either succeeding or failing.
• The installed package was broken if you used npm --ignore-scripts because then the post-install script wasn't run. Some people enable this option so that malicious packages must be run first before being able to do malicious stuff.

The new installation strategy automatically downloads the correct binary executable using npm's optionalDependencies feature to depend on all esbuild packages for all platforms but only have the one for the current platform be installed. This is a built-in part of the package manager so my assumption is that it should work correctly in all of these edge cases that currently don't work. And if there's an issue with this, then the problem is with the package manager instead of with esbuild so this should hopefully reduce the maintenance burden on esbuild itself. Changing to this installation strategy has these drawbacks:

• Old versions of certain package managers (specifically npm and yarn) print lots of useless log messages during the installation, at least one for each platform other than the current one. These messages are harmless and can be ignored. However, they are annoying. There is nothing I can do about this. If you have this problem, one solution is to upgrade your package manager to a newer version.

• Installation will be significantly slower in old versions of npm, old versions of pnpm, and all versions of yarn. These package managers download all packages for all platforms even though they aren't needed and actually cannot be used. This problem has been fixed in npm and pnpm and the problem has been communicated to yarn: [Bug?]: Yarn downloads irrelevant optional dependencies (e.g. for a different OS) yarnpkg/berry#3317. If you have this problem, one solution is to use a newer version of npm or pnpm as your package manager.

• This installation strategy does not work if you use npm --no-optional since then the package with the binary executable is not installed. If you have this problem, the solution is to not pass the --no-optional flag when installing packages.

• There is still a small post-install script but it's now optional in that the esbuild package should still function correctly if post-install scripts are disabled (such as with npm --ignore-scripts). This post-install script optimizes the installed package by replacing the esbuild JavaScript command shim with the actual binary executable at install time. This avoids the overhead of launching another node process when using the esbuild command. So keep in mind that installing with --ignore-scripts will result in a slower esbuild command.

Despite the drawbacks of the new installation strategy, I believe this change is overall a good thing to move forward with. It should fix edge case scenarios where installing esbuild currently doesn't work at all, and this only comes at the expense of the install script working in a less-optimal way (but still working) if you are using an old version of npm. So I'm going to switch installation strategies and see how it goes.

The platform-specific binary executables are still hosted on npm in the same way, so anyone who wrote code that downloads builds from npm using the instructions here should not have to change their code: https://esbuild.github.io/getting-started/#download-a-build. However, note that these platform-specific packages no longer specify the bin field in package.json so the esbuild command will no longer be automatically put on your path. The bin field had to be removed because of a collision with the bin field of the esbuild package (now that the esbuild package depends on all of these platform-specific packages as optional dependencies).

Related issues:

Fixes #286
Fixes #291
Fixes #319
Fixes #347
Fixes #369
Fixes #547
Fixes #565
Fixes #789
Fixes #921
Fixes #1193
Fixes #1270
Fixes #1382
Fixes #1422
Fixes #1450
Fixes #1485
Fixes #1546
Fixes #1547
Fixes #1574
Fixes #1609

[lukeed]

Is there a way to test this? Are you looking for people to test this in their toolchains?

[evanw]

Testing this out would be much appreciated. I have published a test package called esbuild-experimental-optdeps@0.12.10 which should be an accurate representation of how the esbuild package will behave after this PR lands. I have tested this with npm, pnpm, and yarn and everything seems to work.

However, it would be good to know if this works with unusual configurations such as custom registries, HTTP proxies, offline installations, etc. Right now I'm assuming it'll work since optionalDependencies is a long-standing existing feature of the npm ecosystem, but there could always be package manager bugs around this feature. The tracking issue for this PR is #789 so you can read that thread for more details and discussion about this approach.

I'm assuming that asking people to test this out prior to it landing isn't sufficient to flush out any issues with the approach since I've already asked before and very few people have actually tried it out. So as long as there aren't any blocker problems, I'm planning on merging this as a breaking change release to get wider feedback. But testing this out would still be very much appreciated as it's obviously better to uncover blocker issues before it lands.

[lukeed]

Worked for me! I'm running pnpm@5.11.1 and it worked flawlessly in a regular repo and a monorepo environment (with multiple packages depending on esbuild-experimental-optdeps@0.12.10 directly).

My concern was that it was going to an experience similar to @swc/core installation, which downloads all binaries and then only keeps the matching package, but that wasn't the case 🎉

[hardfist]

Worked for me! I'm running pnpm@5.11.1 and it worked flawlessly in a regular repo and a monorepo environment (with multiple packages depending on esbuild-experimental-optdeps@0.12.10 directly).

My concern was that it was going to an experience similar to @swc/core installation, which downloads all binaries and then only keeps the matching package, but that wasn't the case 🎉

pnpm has already fixes the bug since latest v6 verison and downloads only partial binaries of @swc/core

[lukeed]

@hardfist right, but I wanted to see if the non-patched version treated this PR the same way

[danvim]

Can confirm this (0.13.2) is working for me behind proxy. Coming from #319 🥳

I no longer need to prime the installation process by pre-downloading the latest version without the --prefer-offline option if I had previously installed esbuild at an older version.

[silverwind]

Can also confirm this as working in cases where it previously failed with a network timeout trying to contact the registry directly. 🎉