build,tools: make addons tests work with GN

[zcbenz]

This PR refactors the install and build_addons scripts so they can be reused by node-ci and electron projects to build the addons tests.

In detail, this PR does following things:

1. Minor modifications of the GN configurations to fix loading native modules.
2. Add some flags to tools/install.py script to allow customize locations of config.gypi, v8, etc. With some refactoring to remove global variables.
3. Rewrite the tools/build_addons.py script:
   1. Rewrite it from js to python, because implementing cmd args handling with vanilla Node.js would take much longer time than a simple rewrite.
   2. Add a few more flags to make it easier to use for embedders.
4. Modify the Makefile to generate headers in out dir first, and then calls build_addons.py --headers-dir=out_dir/headers (which calls node-gyp --nodedir=out_dir/headers) to build addons. Previously the build system was just calling node-gyp --nodedir=., which relied on quirky behavior of node-gyp and did not really test whether the generated headers work.

Users that rely on make or vcbuild.bat to work on Node are not affected.

[nodejs-github-bot]

Review requested:

• @nodejs/tsc

[zcbenz]

I have separated some changes out of this PR so review should be easier.

/cc @nodejs/gyp @joyeecheung @richardlau for reviewing changes on tools.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/55698/

[zcbenz]

Ping @nodejs/gyp @joyeecheung @anonrig for review after the holiday.

[joyeecheung]

LGTM % a correction but I think ideally we should have someone from @nodejs/build to confirm that it works with our releases.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/56066/

[nodejs-github-bot]

Commit Queue failed

- Loading data for nodejs/node/pull/50737
✔  Done loading data for nodejs/node/pull/50737
----------------------------------- PR info ------------------------------------
Title      build,tools: make addons tests work with GN (#50737)
   ⚠  Could not retrieve the email or name of the PR author's from user's GitHub profile!
Branch     zcbenz:build-addons-gn -> nodejs:main
Labels     windows, build, openssl, libuv, tools, python, needs-ci, dependencies, commit-queue-squash
Commits    3
 - build,tools: make addons tests work with GN
 - build,tools: generate node headers on request
 - build,tools: there are 2 out dirs on windows
Committers 1
 - Cheng Zhao 
PR-URL: https://github.com/nodejs/node/pull/50737
Reviewed-By: Joyee Cheung 
Reviewed-By: Benjamin Gruenbaum 
Reviewed-By: Yagiz Nizipli 
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/50737
Reviewed-By: Joyee Cheung 
Reviewed-By: Benjamin Gruenbaum 
Reviewed-By: Yagiz Nizipli 
--------------------------------------------------------------------------------
   ⚠  Commits were pushed since the last approving review:
   ⚠  - build,tools: make addons tests work with GN
   ⚠  - build,tools: generate node headers on request
   ⚠  - build,tools: there are 2 out dirs on windows
   ℹ  This PR was created on Wed, 15 Nov 2023 08:29:46 GMT
   ✔  Approvals: 3
   ✔  - Joyee Cheung (@joyeecheung) (TSC): https://github.com/nodejs/node/pull/50737#pullrequestreview-1810123695
   ✔  - Benjamin Gruenbaum (@benjamingr) (TSC): https://github.com/nodejs/node/pull/50737#pullrequestreview-1756204771
   ✔  - Yagiz Nizipli (@anonrig) (TSC): https://github.com/nodejs/node/pull/50737#pullrequestreview-1805222548
   ✔  Last GitHub CI successful
   ℹ  Last Full PR CI on 2024-02-23T01:17:58Z: https://ci.nodejs.org/job/node-test-pull-request/57320/
- Querying data for job/node-test-pull-request/57320/
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  Aborted `git node land` session in /home/runner/work/node/node/.ncu

https://github.com/nodejs/node/actions/runs/8015803850

[zcbenz]

Landed in b1468d2

[batrla]

Sorry to report it, but this PR caused regression, build in --shared mode now fails:

11:02:41: stdout installing /some_dir/build/prototype/i386/some_bindir/node
11:02:41: stderr Traceback (most recent call last):
11:02:41: stderr   File "/some_dir/build/amd64/tools/install.py", line 419, in <module>
11:02:41: stderr     run(parse_options(sys.argv[1:]))
11:02:41: stderr   File "/some_dir/build/amd64/tools/install.py", line 372, in run
11:02:41: stderr     files(options, install)
11:02:41: stderr   File "/some_dir/build/amd64/tools/install.py", line 179, in files
11:02:41: stderr     action(options, [os.path.join(output_prefix, output_lib)],
11:02:41: stderr                                   ^^^^^^^^^^^^^
11:02:42: stderr UnboundLocalError: cannot access local variable 'output_prefix' where it is not associated with a value   

The problem in source code is obvious, the variable 'output_prefix' is only defined in some branches.
Later in code the variable is used in a branch where it was not defined before.

[zcbenz]

Sorry for breaking it! I'll create a fix soon.