Allow building CPython from a local source directory

[zanieb]

As a first step towards #139, adds the ability to build from a local source directory instead of downloading CPython from python.org via a new --python-source <path> build option.

e.g. tested by cloning CPython at tag v3.12.1 then building

PYBUILD_PYTHON_VERSION=3.12.1 python3.12 ./build-macos.py \
    --target-triple aarch64-apple-darwin \
    --python cpython-3.12 --optimizations debug \
    --python-source $(pwd)/../cpython

I also tested this with the latest commit on the CPython 3.12 branch.

There are a lot of assumptions about build information being available in the DOWNLOADS file, which makes this a little brittle. I also only implemented this for Unix.

I tried to make as minimal of a change as possible i.e. instead of refactoring to allow an unpacked source directory to be passed around we put the given source in an archive simulating a download. We could follow this with some more changes to abstract Python source downloads.

Additional safeguards could be added to avoid common mistakes, like ensuring that:

• The PYBUILD_PYTHON_VERSION version is compatible with the --python version
• The --python-source path exists
• Relative --python-source paths are resolved based on the original invocation working directory

Additionally, we could improve usability by:

• Inferring the PYBUILD_PYTHON_VERSION from the source files
• Allowing the source to be a remote path e.g. from GitHub
• Read commit information from Git and include in builds

Next steps for testing in-progress CPython builds would involve:

• Setting up a CI job to pull and build from Python 3.12's development branch
• Initial Python 3.13 support
• Similar changes to the Windows builds

[zanieb]

Sorry, some of these look like format-on-save. I can revert.

[indygreg]

I thought I had black formatting on for all the Python files in this repo. It is possible I haven't picked up a modern version of black yet.

I'd welcome PRs to integrate ruff :)

[zanieb]

I'll open a pull request to add Ruff :D

[indygreg]

This approach seems pretty reasonable.

A potential alternative worth considering is using the docker -v functionality to bind mount the source checkout into the build environment. You would need to hack up build-cpython.sh to copy the source files into the /build directory. I think always normalizing to a source archive makes a lot of sense.

I agree it would be nice to have smarter version resolution or Git (both local and remote repo) integration. But those could be added later.

I'm sure you figured this out, but the build-main.py script and the way we use PYBUILD_* environment variables to thread state around is super hacky. Evolved complexity. If you have ideas for simplifying, I'm all ears. Maybe we could integrate rye or uv to make bootstrapping the build environment simpler?

[indygreg]

I thought I had black formatting on for all the Python files in this repo. It is possible I haven't picked up a modern version of black yet.

I'd welcome PRs to integrate ruff :)

[zanieb]

@indygreg I'm blocked by #238 on verifying that we can use this new flag in Linux, but I made sure everything works in CI in my fork.

A potential alternative worth considering is using the docker -v functionality to bind mount the source checkout into the build environment. You would need to hack up build-cpython.sh to copy the source files into the /build directory. I think always normalizing to a source archive makes a lot of sense.

Yeah the normalization makes sense to me for now. It's not really a bottleneck.

I agree it would be nice to have smarter version resolution or Git (both local and remote repo) integration. But those could be added later.

I can open issues to track those.

I'm sure you figured this out, but the build-main.py script and the way we use PYBUILD_* environment variables to thread state around is super hacky. Evolved complexity. If you have ideas for simplifying, I'm all ears.

I'm not ready to make any drastic changes, but I'll keep it in mind. I actually didn't think it was too bad.

Maybe we could integrate rye or uv to make bootstrapping the build environment simpler?

Honestly I was impressed by your bootstrapping :) I'd definitely be willing to add uv though.

[zanieb]

Just to check in here again, is there something you are looking for before this can be merged? Should I just stack another pull request that starts to make use of this? Since main is moving slow I don't mind building up a stack showing functionality before merging.

[indygreg]

This looks good. I'm waiting for it to pass CI (I pushed a branch out-of-band) before I merge.