PEP 632: Remove the distutils package (Python 3.12) #92584

vstinner · 2022-05-09T22:42:26Z

The distutils module was deprecated in Python 3.10 and is scheduled for removal in Python 3.12.

Right now, Python still uses distutils for different purpose:

Build C extensions which can not be built by Makefile yet. Python 3.11 can build way more C extensions with Makefile than Python 3.10.
peg_generator uses distutils to build C extensions.
test_cppext uses distutils to check that the Python C API can be used in C++.
Tools/c-analyzer/ uses distutils to get a C preprocessor.

Until Python can be built and used without distutils, I propose to first rename the distutils package to _distutils: rename th Lib/distutils/ directory to Lib/_distutils/.

Right now, test_venv fails without distutils, because pip fails on importing the distutils module in pip/_internal/locations/_distutils.py: https://github.com/pypa/pip/blob/cb24fb4052ca8ab8009866b0de61980c81a7e13c/src/pip/_internal/locations/_distutils.py#L9-L12

I created the PR #92585 which renames distutils to _distutils.

PR: gh-92584: Remove the distutils package #99061

PR: gh-92584: Remove references to removed _bootsubprocess #99062

PR: gh-100238: Vendor distutils for peg_generator #104744

PR: GH-92584: Move installation schemes overview to sysconfig docs #108018

PR: GH-92584: Remove Installing Python Modules (Distutils version) #108020

PR: GH-92584: Remove distutils from the newtypes tutorial includes #108024

PR: GH-92584: Remove distutils mentions from importlib.metadata docs #108026

PR: GH-92584: Redirect to Setuptools' guidance on extension modules #108030

PR: GH-92584: Drop reference to Distutils in site.USER_BASE #108031

PR: [3.12] GH-92584: Drop reference to Distutils in site.USER_BASE (GH-108031) #108039

PR: GH-92584: Remove references to Distutils in PYTHONUSERBASE #108040

PR: GH-92584: Remove references to Distutils in configure.rst #108043

PR: GH-92584: Redirect macOS package installation to the PPUG #108044

PR: GH-92584: Remove reference to Distutils in cx_Freeze's description #108047

PR: [3.12] GH-92584: Remove reference to Distutils in cx_Freeze's description (GH-108047) #108057

PR: [3.12] GH-92584: Redirect macOS package installation to the PPUG (GH-108044) #108058

PR: [3.11] GH-92584: Redirect macOS package installation to the PPUG (GH-108044) #108059

PR: [3.12] GH-92584: Remove references to Distutils in PYTHONUSERBASE (GH-108040) #108060

PR: [3.11] GH-92584: Remove reference to Distutils in cx_Freeze's description (GH-108047) #108061

PR: [3.12] GH-92584: Remove Installing Python Modules (Distutils version) (GH-108020) #108062

PR: [3.12] GH-92584: Remove references to Distutils in configure.rst (GH-108043) #108063

PR: [3.12] GH-92584: Remove distutils from the newtypes tutorial includes (GH-108024) #108333

PR: [3.12] GH-92584: Move installation schemes overview to sysconfig docs (GH-108018) #110213

PR: [3.11] GH-92584: Move installation schemes overview to sysconfig docs (GH-108018) #110214

The text was updated successfully, but these errors were encountered:

vstinner · 2022-05-09T22:48:29Z

Right now, test_venv fails without distutils, because pip fails on importing the distutils module in pip/_internal/locations/_distutils.py

I reported this issue to pip: pypa/pip#11103

vstinner · 2022-05-09T22:53:16Z

The distutils package was deprecated by the issue #85454.

vstinner · 2022-05-10T00:09:23Z

PEP 632 "Deprecate distutils module": https://peps.python.org/pep-0632/

FFY00 · 2022-05-10T00:40:08Z

That seems like a good plan. Thanks.

jaraco · 2022-05-10T11:16:59Z

My fear with renaming it is that it lingers indefinitely, and diverging in subtle ways from pypa/distutils. Why not address the dependencies and fully remove the functionality?

Regardless, I agree renaming is preferable to not removing the module at all.

vstinner · 2022-05-10T11:20:15Z

Why not address the dependencies and fully remove the functionality?

Someone has to modify Python to no longer depend on distutils. The issue description lists code using it. It doesn't seem trivial.

FFY00 · 2022-05-10T14:01:58Z

Would it be reasonable to keep it around in the repo while we don't drop the dependency but not ship it as part of Python installations?

vstinner · 2022-05-10T15:23:24Z

Would it be reasonable to keep it around in the repo while we don't drop the dependency but not ship it as part of Python installations?

With my PR, the Lib/_distutils/ directory is not installed by make install.

jkloth · 2022-05-10T15:56:16Z

Do note that we have buildbots that run from an installed Python. The tests would need to also be modified to fail gracefully
(or skip) when _distutils is not available.

vstinner · 2022-05-10T16:01:16Z

Do note that we have buildbots that run from an installed Python. The tests would need to also be modified to fail gracefully (or skip) when _distutils is not available.

I propose skipping tests relying on _distutils if _distutils is not available.

vstinner · 2022-05-10T20:53:26Z

I wrote two PRs to avoid the deprecated distutils module in two tests:

gh-92584: test_cppext uses setuptools #92639: test_cppext uses setuptools
gh-92584: test_decimal uses shutil.which() #92640: test_decimal uses shutil.which() -- IMO an uncontroversial change

test_decimal now uses shutil.which() rather than deprecated distutils.spawn.find_executable().

Rewrite test_cppext to run in a virtual environment and to build the C++ extension with setuptools rather than distutils.

AbhigyanBose · 2022-05-15T12:44:47Z

Hi @vstinner

I think #92639 is causing a test failure, details here #92820

PS: Commented here as I thought those changes are related to this PEP.

encukou · 2022-06-23T12:53:13Z

Marking as 3.12.0a1 release blocker. We want to have public-facing distutils gone in the first alpha, to have as much time as possible to deal with the fallout.

tiran · 2022-07-07T12:37:01Z

Actually we can get rid of distutils entirely. I have a working PR that gets rid of our setup.py and builds all extension modules with configure and make.

pradyunsg · 2022-07-08T16:19:27Z

For the curious, the PR that @tiran is refering to is #94474

`make test_cppmods` compiles C++ test extension modules.

…H-108031) (#108039) GH-92584: Drop reference to Distutils in ``site.USER_BASE`` (GH-108031) Drop reference to Distutils in ``site.USER_BASE`` (cherry picked from commit f2a9dfd) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

…cription (GH-108047) (#108057) GH-92584: Remove reference to Distutils in ``cx_Freeze``'s description (GH-108047) Remove reference to Distutils in ``cx_Freeze``'s description (cherry picked from commit 57fcf96) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

…GH-108040) (#108060) GH-92584: Remove references to Distutils in ``PYTHONUSERBASE`` (GH-108040) Remove references to Distutils in ``PYTHONUSERBASE`` (cherry picked from commit 636ca31) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

…108044) (#108058) GH-92584: Redirect macOS package installation to the PPUG (GH-108044) (cherry picked from commit 9028642) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

…108043) (#108063) GH-92584: Remove references to Distutils in configure.rst (GH-108043) Remove references to Distutils in configure.rst (cherry picked from commit e88eb37) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

…GH-108020) (#108062) * [3.12] GH-92584: Remove Installing Python Modules (Distutils version) (GH-108020). (cherry picked from commit fbb7cbc) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

…ythonGH-108024) (cherry picked from commit e97b7be) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

…GH-108024) (#108333) GH-92584: Remove distutils from the newtypes tutorial includes (GH-108024) (cherry picked from commit e97b7be) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

The package `distutils` is deprecated since python 3.10 and it is scheduled to be removed with python 3.12, see: https://docs.python.org/3.12/whatsnew/3.12.html python/cpython#92584 `shutil.copytree()` is able to copy the source directory tree to an existing destination directory only from python 3.8 (parameter `dirs_exist_ok=True`), for python < 3.8 still use `distutils.dir_util.copy_tree()`

The package `distutils` is deprecated since python 3.10 and it is scheduled to be removed with python 3.12, see: https://docs.python.org/3.12/whatsnew/3.12.html python/cpython#92584 `shutil.copytree()` is able to copy the source directory tree to an existing destination directory only from python 3.8 (parameter `dirs_exist_ok=True`), for python < 3.8 still use `distutils.dir_util.copy_tree()` Raise worker version to 214 (also server side)

The package `distutils` is deprecated since python 3.10 and it is scheduled to be removed with python 3.12, see: https://docs.python.org/3.12/whatsnew/3.12.html python/cpython#92584 `shutil.copytree()` is able to copy the source directory tree to an existing destination directory only from python 3.8 (parameter `dirs_exist_ok=True`), for python < 3.8 still use `distutils.dir_util.copy_tree()` While here: - create in a safe way the temporary directory used for the update - deal with possible download problems Raise worker version to 214 (also server side)

* Add new installation path functions subsection * Add content from install/index to sysconfig * Fix table * Update note about installers * Clean up the list of schemes, remove references to Distutils

…ythonGH-108018) * Add new installation path functions subsection * Add content from install/index to sysconfig * Fix table * Update note about installers * Clean up the list of schemes, remove references to Distutils (cherry picked from commit f16e81f) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

…g docs (pythonGH-108018) * Add new installation path functions subsection * Add content from install/index to sysconfig * Fix table * Update note about installers * Clean up the list of schemes, remove references to Distutils. (cherry picked from commit f16e81f) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

…GH-108018) (#110213) GH-92584: Move installation schemes overview to sysconfig docs (GH-108018) * Add new installation path functions subsection * Add content from install/index to sysconfig * Fix table * Update note about installers * Clean up the list of schemes, remove references to Distutils (cherry picked from commit f16e81f) Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

…GH-108018) (#110214) * Add new installation path functions subsection * Add content from install/index to sysconfig * Fix table * Update note about installers * Clean up the list of schemes, remove references to Distutils. (cherry picked from commit f16e81f)

…ython#108018) * Add new installation path functions subsection * Add content from install/index to sysconfig * Fix table * Update note about installers * Clean up the list of schemes, remove references to Distutils

vstinner added the type-bug An unexpected behavior, bug, or error label May 9, 2022

vstinner changed the title ~~Remove the distutils package to _distutils~~ Rename the distutils package to _distutils May 9, 2022

vstinner mentioned this issue May 9, 2022

pip imports the deprecated distutils module pypa/pip#11103

Closed

1 task

vstinner mentioned this issue May 9, 2022

Deprecate and remove distutils #85454

Closed

vstinner mentioned this issue May 10, 2022

[WIP] gh-92584: Rename the distutils package to _distutils #92585

Closed

vstinner changed the title ~~Rename the distutils package to _distutils~~ PEP 632: Rename the distutils package to _distutils May 10, 2022

vstinner added a commit that referenced this issue May 10, 2022

gh-92584: test_decimal uses shutil.which() (#92640)

dfdebda

test_decimal now uses shutil.which() rather than deprecated distutils.spawn.find_executable().

vstinner added a commit that referenced this issue May 12, 2022

gh-92584: test_cppext uses setuptools (#92639)

4e28377

Rewrite test_cppext to run in a virtual environment and to build the C++ extension with setuptools rather than distutils.

ericsnowcurrently mentioned this issue May 25, 2022

gh-202: Simplify Windows installation python/pyperformance#203

Open

encukou added the release-blocker label Jun 23, 2022

pablogsal added the 3.12 bugs and security fixes label Jun 23, 2022

gvanrossum assigned zooba Jul 3, 2022

vstinner changed the title ~~PEP 632: Rename the distutils package to _distutils~~ PEP 632: Rename the distutils package to _distutils (Python 3.12) Jul 7, 2022

tiran added a commit to tiran/cpython that referenced this issue Aug 5, 2022

pythongh-92584: Add Makefile rules for test cpp extensions

f121f28

`make test_cppmods` compiles C++ test extension modules.

bedevere-bot mentioned this issue Aug 5, 2022

gh-92584: Add Makefile rules for test cpp extensions #95721

Draft

ambv pushed a commit that referenced this issue Aug 22, 2023

GH-92584: Remove distutils from the newtypes tutorial includes (#108024)

e97b7be

bedevere-bot mentioned this issue Aug 22, 2023

[3.12] GH-92584: Remove distutils from the newtypes tutorial includes (GH-108024) #108333

Merged

ppigazzini mentioned this issue Aug 28, 2023

Replace deprecated distutils package official-stockfish/fishtest#1771

Merged

bedevere-app bot mentioned this issue Oct 2, 2023

[3.12] GH-92584: Move installation schemes overview to sysconfig docs (GH-108018) #110213

Merged

bedevere-app bot mentioned this issue Oct 2, 2023

[3.11] GH-92584: Move installation schemes overview to sysconfig docs (GH-108018) #110214

Merged

robertgro mentioned this issue Nov 17, 2023

Update to Python 3.12 - Remove 'stdlib distutils module' requirement (deprecated) marella/chatdocs#97

Open

santaimpersonator mentioned this issue Dec 13, 2023

Python 3.12 Compatibility backstage/mkdocs-monorepo-plugin#117

Closed

Uzume mentioned this issue May 18, 2024

Python 3.13 support yaml/pyyaml#797

Closed

7 tasks

ferdnyc mentioned this issue Jun 16, 2024

Python version update wireviz/WireViz#309

Open

martinrieder mentioned this issue Jul 13, 2024

Processing Jumpers and Loops, Plus Processing Wires inside the Table as Edges wireviz/WireViz#369

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PEP 632: Remove the distutils package (Python 3.12) #92584

PEP 632: Remove the distutils package (Python 3.12) #92584

vstinner commented May 9, 2022 •

edited by bedevere-app bot

Loading

vstinner commented May 9, 2022

vstinner commented May 9, 2022

vstinner commented May 10, 2022

FFY00 commented May 10, 2022

jaraco commented May 10, 2022

vstinner commented May 10, 2022

FFY00 commented May 10, 2022

vstinner commented May 10, 2022

jkloth commented May 10, 2022

vstinner commented May 10, 2022

vstinner commented May 10, 2022

AbhigyanBose commented May 15, 2022 •

edited

Loading

encukou commented Jun 23, 2022

tiran commented Jul 7, 2022

pradyunsg commented Jul 8, 2022

PEP 632: Remove the distutils package (Python 3.12) #92584

PEP 632: Remove the distutils package (Python 3.12) #92584

Comments

vstinner commented May 9, 2022 • edited by bedevere-app bot Loading

vstinner commented May 9, 2022

vstinner commented May 9, 2022

vstinner commented May 10, 2022

FFY00 commented May 10, 2022

jaraco commented May 10, 2022

vstinner commented May 10, 2022

FFY00 commented May 10, 2022

vstinner commented May 10, 2022

jkloth commented May 10, 2022

vstinner commented May 10, 2022

vstinner commented May 10, 2022

AbhigyanBose commented May 15, 2022 • edited Loading

encukou commented Jun 23, 2022

tiran commented Jul 7, 2022

pradyunsg commented Jul 8, 2022

vstinner commented May 9, 2022 •

edited by bedevere-app bot

Loading

AbhigyanBose commented May 15, 2022 •

edited

Loading