build: Add new build infrastructure

[adisbladis]

This fixes many problems existing in the Nixpkgs Python infrastructure:

• No dependency propagation

No leaking of dependencies via PYTHONPATH

• No runtime deps at build time

This causes much less rebuilds

• Support for circular dependencies

• No wrapping of Python bin's

No strange venv breaking shell wrapper

• Proper virtualenvs only

Meaning that applications that spawns children using sys.executable works fine.

• Hermetic bootstrap

Nixpkgs breaks if you override any bootstrap dependencies with wheels.

• Manual cross splicing

Meaning you can override build-time and runtime deps separately, and also IMHO just in general a much better cross interface.

---

I've previously tried to tackle this in NixOS/nixpkgs#272178.
This is not having the same backwards compat concerns.

[adisbladis]

Example usage:

{ pyproject-nix, pkgs }:

let
  python = pkgs.python312;

  # Inject your own packages on top with overrideScope
  pythonSet = pkgs.callPackage pyproject-nix.build.packages {
    inherit python;
  };

in pythonSet.pythonPackagesHostHost.mkVirtualEnv "test-venv" {
  build = [ ];
}

[TyberiusPrime]

This is great work, and I especially like the recent symlinks fixes.

One case where we still run into a collision in bin (=needs dependency ordering) is
"fastapi[standard]==0.114.0", which leads to

  > error: collision between `/nix/store/caq1nnyypsr16yrb4j6cpwxxyvcxd4rd-python3.12-fastapi-cli-0.0.5/bin/fastapi' and `/nix/store/722hdr0lwkzj3qdppr39r35vnrxvmjys-python3.12-fastapi-0.114.0/bin/fastapi'

[adisbladis]

This is great work, and I especially like the recent symlinks fixes.

One case where we still run into a collision in bin (=needs dependency ordering) is "fastapi[standard]==0.114.0", which leads to

  > error: collision between `/nix/store/caq1nnyypsr16yrb4j6cpwxxyvcxd4rd-python3.12-fastapi-cli-0.0.5/bin/fastapi' and `/nix/store/722hdr0lwkzj3qdppr39r35vnrxvmjys-python3.12-fastapi-0.114.0/bin/fastapi'

Are you sure you're using the right builder? This error message looks to originate in buildEnv, which is not used to construct the virtualenv.

[TyberiusPrime]

This is great work, and I especially like the recent symlinks fixes.
One case where we still run into a collision in bin (=needs dependency ordering) is "fastapi[standard]==0.114.0", which leads to

  > error: collision between `/nix/store/caq1nnyypsr16yrb4j6cpwxxyvcxd4rd-python3.12-fastapi-cli-0.0.5/bin/fastapi' and `/nix/store/722hdr0lwkzj3qdppr39r35vnrxvmjys-python3.12-fastapi-0.114.0/bin/fastapi'

Are you sure you're using the right builder? This error message looks to originate in buildEnv, which is not used to construct the virtualenv.

My bad, that was the old builder.

The current builder gives

      > Executing pyprojectMakeVenv
       > Traceback (most recent call last):
       >   File "/nix/store/k44y6ylpzvk62wrz1s72gq7ym6z0dx1r-make-venv.py", line 216, in <module>
       >     main()
       >   File "/nix/store/k44y6ylpzvk62wrz1s72gq7ym6z0dx1r-make-venv.py", line 212, in main
       >     link_dependency(dep_root, out_root)
       >   File "/nix/store/k44y6ylpzvk62wrz1s72gq7ym6z0dx1r-make-venv.py", line 136, in link_dependency
       >     _link(dep_root, out_root)
       >   File "/nix/store/k44y6ylpzvk62wrz1s72gq7ym6z0dx1r-make-venv.py", line 119, in _link
       >     write_bin_dir(path, out.joinpath("bin"))
       >   File "/nix/store/k44y6ylpzvk62wrz1s72gq7ym6z0dx1r-make-venv.py", line 42, in write_bin_dir
       >     with open(bin_file_out, "wb") as f_out:
       >          ^^^^^^^^^^^^^^^^^^^^^^^^
       > PermissionError: [Errno 13] Permission denied: '/nix/store/4kzghrgzklaajq792djhx8dv6cxja9ff-test-venv/bin/fastapi'

[adisbladis]

Merging this now documented as an experimental feature.