Naming scheme for free-threaded Python

[native-api]

Subj, continuing discussion in #2995 and #2996 .

• First of all -- what are the upstream's plans for free-threaded Python? Do they plan to make separate releases for it? If yes -- how do they plan to name them -- at least, internally, at the download site?
• Then, the naming scheme must not break pyenv latest. So
  • either it's going to be a prefix (and we need to come up with one).
  • or we'll need to introduce (and document) a change to pyenv latest to treat specific parts of the name specially
    • this may not be just a special case -- since we do already treat some prefixes specially (e.g. -dev, -src, aX/bX).

[hugovk]

• First of all -- what are the upstream's plans for free-threaded Python? Do they plan to make separate releases for it? If yes -- how do they plan to name them -- at least, internally, at the download site?

Upstream, the official macOS and Windows installers have an option to install the free-threading binaries, which also installs python3.13t alongside the regular python3.13:

• Provide downloadable free-threaded macOS builds for Python 3.13 python/cpython#120098
• https://docs.python.org/3.13/using/windows.html#installing-free-threaded-binaries

So for example, there's a single 3.13.0b3 release with a single macOS installer. When running the installer you can choose whether to install free-threaded or regular:

• https://www.python.org/downloads/release/python-3130b3/

cc @colesbury

[native-api]

which also installs python3.13t alongside the regular python3.13

Interesting. So, in Windows and MacOS, you can install both versions alongside each other -- but in Linux, you can only choose one or the other?

[colesbury]

You can install them side by side on Linux too, although there's this bug. Windows and macOS have unified installers -- Linux doesn't really have an installer, just various package managers.

[native-api]

I'm hinting that if the upstream's intention is to ship them together in the same installation then we should probably do so, too.

Reading https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional-in-cpython-acceptance/37075 however, the current upstream's intention (Phase I) seems to be that this feature is experimental:

Phase I: Experimental phase, which can start immediately, in which the free-threaded build is enabled through a build-time option. This should not be the default install anywhere. At least one major Python release should include this experimental free-threaded build, to allow third-party packages to test and do their own experimentation. In this stage we should make it clear the build is experimental, not supported for “production use”, and may be reverted.

Which for us seems to clearly mean "a separate installation".

[hugovk]

When you run the installer (for macOS, or for Windows), the default is the regular GIL build.

However, during installation, you can select an option and install the free-threaded build instead.

You don't get both GIL and free-threaded builds installed by default.

But if you like, you can run the installer a second time and choose the other one, and have both installed side-by-side (noting the bug above).

[native-api]

As mentioned above, the upstream's plan regarding free-threaded builds is laid out at https://discuss.python.org/t/pep-703-making-the-global-interpreter-lock-optional-in-cpython-acceptance/37075 as follows:

• Phase I: Experimental phase, which can start immediately, in which the free-threaded build is enabled through a build-time option. This should not be the default install anywhere. At least one major Python release should include this experimental free-threaded build, to allow third-party packages to test and do their own experimentation. In this stage we should make it clear the build is experimental, not supported for “production use”, and may be reverted.
• Phase II: Supported-but-not-default phase, which would start when the API and ABI changes have sufficiently settled, and there is sufficient community support. Exact criteria for this phase are hard to pin down at this stage, so this will have to involve some discussion among Core Devs and the community, and a decision by the SC at the time. At this point reverting should still be possible (so for example preprocessor guards should remain in place) although obviously we aren’t expecting it.
• Phase III: Default phase, at which point the free-threaded build becomes the default (but can initially still be disabled). Again, the exact criteria are hard to pin down this far ahead, but the aim is to make this as seamless and painless a default flip as possible. Like the previous phase, the SC at the time will need to make a decision as to when this occurs. Some time after the default flip, when we have a good indication it’s no longer widely used, we should start the discussion on removing the GIL build entirely.

---

Which for us means that

• During Phase I, there'll be a run of separate free-threaded releases for at least one major CPython version
  • since it'll be a run, we probably have to support resolving among them in pyenv latest
• During Phase II, we'll probably continue shipping the same practice
  • or we'll start shipping them together in the same (main) release if the build supports that -- but since it's "not the default", the python executable will be the GIL version.
    • changes to the after-build aliasing procedure will be needed to make separate aliases for t executables.
    • this path would mean that the "separate-release" run would be short-lived which makes supporting it in pyenv-latest questionable
• During Phase III, the default will switch
  • if there will be combined releases, this probably means further modifications to the aliasing procedure
    • combined with the above, this suggests that combined releases will bring sizeable complications, so I guess we're not doing them unless there's strong demand for this specific format
  • not sure if we should continue shipping GIL releases for some time under some separate alias. There probably will be some demand, so we'll probably do that for at least one major version
    • in this case, if we support resolving suffixed versions, we'll need to support this suffix as well. As for the suffix, the upstream will probably come up with one for us just the same 😉
  • Also not sure how pyenv latest should handle the switch
    • Should it "intelligently" (read: implicitly and confusingly) resolve "t" to regular Phase III releases?
      • Probably not, we'd rather have users change their prefixes. Transitions through breaking changes virtually always require manual intenventions, so we would likely do a better service to our users by making them do the switch explicitly

---

Concluding the above, the upstream's course seems to suggest that the need to support resolving the "t"-suffixed versions in pyenv latest -- maybe not right now but at least by the time an upstream release is made.