New venv launchers are broken

[naveen521kk]

Got the recent build from staging created the virtual environment with python3.10. The venv was created as expected. Then I ran the executable bin/python.exe and it launched python 3.9 (??). I then set and an env var to debug those launcher PYLAUNCH_DEBUG=1 and ended up with these logs

❯ .venv-3.10/bin/python --version
launcher build: 32bit
launcher executable: Console
File 'C:\dev\temp-test\.venv-3.10\bin\pyvenv.cfg' non-existent
Using venv configuration file 'C:\dev\temp-test\.venv-3.10\pyvenv.cfg'
Called with command line: --version
run_child: about to run 'C:/msys64/mingw32/bin\python.exe --version'
Python 3.9.10
child process exit code: 0

It launched the python.exe instead of python3.10.exe. This is the first broken behaviour I noticed.

Then I copied python3.10.exe to python.exe and then recreated that virtual environment. While recreating it ended up running the ensurepip module (the one that bootstraps pip and setuptools) globally and not in the virtual environment

❯ python -m venv .venv-3.10
launcher build: 32bit
launcher executable: Console
File 'C:\dev\temp-test\.venv-3.10\bin\pyvenv.cfg' non-existent
Using venv configuration file 'C:\dev\temp-test\.venv-3.10\pyvenv.cfg'
Called with command line: -Im ensurepip --upgrade --default-pip
run_child: about to run 'C:/msys64/mingw32/bin\python.exe -Im ensurepip --upgrade --default-pip'
Looking in links: c:\Users\naveen\AppData\Local\Temp\tmp_dm1_pvu
Processing c:\users\naveen\appdata\local\temp\tmp_dm1_pvu\setuptools-58.1.0-py3-none-any.whl
Processing c:\users\naveen\appdata\local\temp\tmp_dm1_pvu\pip-21.2.4-py3-none-any.whl
Installing collected packages: setuptools, pip
Successfully installed pip-21.2.4 setuptools-58.1.0
child process exit code: 0

This is the second broken behaviour I noticed.

Next I ran bin/python -c 'import sys;print(sys.executable)' and it ended up printing

❯ .venv-3.10/bin/python -c 'import sys; print(sys.executable)'
C:/msys64/mingw32/bin\python.exe

which is again wrong, because it should be printing the virtual environment's executable and not the original one. This is the third broken behaviour I noticed.

Now, the last one made me think that somewhere our python can't detect it's a venv when launched through those launchers, but it could detect it when those executables are copied. I looked into where sys.executable is set (it's in Modules/getpath.c) and found that the file is only for UNIX like system and MSVC python uses PC/getpathp.c which I think is the root cause of the problems (from this comment below).

cpython-mingw/Modules/getpath.c

Line 104 in 1a4e816

* NOTE: Windows MSVC builds use PC/getpathp.c instead!

I think the solutions for these problems requires more work, essentially copying stuff from PC/getpathp.c into Modules/getpath.c for supporting these new launchers. Or maybe use PC/getpathp.c rather than just Modules/getpath.c, but it still requires work to move patches from the latter. Also, upstream in Python 3.11 ended up changing these C modules to Python python#29041, so the work we would do would be wasted when upgrading to Python 3.11. Not sure what is the best thing to do here.

Also, I think those patches from #82 should be reverted until these broken behaviours are fixed.

[lazka]

Thanks for testing.

Yes, looks like reverting is the best option at this point.

[naveen521kk]

Reverted msys2/MINGW-packages#11035

what do you think we should do? Wait for Python 3.11 and then apply this patch there, or try and add support for these venv launchers in current versions?

[naveen521kk]

This is fixed by #139