[BUG] Some packages still needing setup.py fail to build in 74.0.0

[Mindstan]

setuptools version

74.0.0

Python version

3.9.18

OS

Ubuntu 22.04.4 x86-64

Additional environment information

The package to install has binary libraries that are already compiled.
pip==24.2, wheel==0.44.0, CPython 3.9.18 compiled with shared libraries option
It worked fine with setuptools<74.
The command is running in a Docker image.

Description

While building a legacy Python package using the command python setup.py install, the process crashed with the following error:

TypeError: object of type 'PosixPath' has no len()

Expected behavior

The package build and install should be successfully completed.

How to Reproduce

This is the setup.py of the packages (with the package name changed). Please note that it is generated by SWIG from a C++ library.

# Builds the extensions with separate SWIG run-time support
# (see http://www.swig.org/Doc1.3/Modules.html) so as to
# share datatypes among different extensions.

import platform, os, sys
from pathlib import Path, PurePath
from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext

if '--help' in str(sys.argv):
    print("""
WARNING! It is no longer supported, neither by The Software nor by Python itself, to run setup.py directly.
Please use `python -m build` and `python -m pip .` for building and installing the Python wrapper instead.
Not specifying any of the options below is not guaranteed to work!

The Software specific options (always as first argument):
  return_software_version

Example:
  python setup.py return_software_version
will print something like:
  2.49.0
""")
    sys.exit()

if "THE_SOFTWARE_SOURCE_DIR" in os.environ:
    the_software_dir = Path(os.environ['THE_SOFTWARE_SOURCE_DIR'])
elif "THE_SOFTWARE_DIR" in os.environ:
    the_software_dir = Path(os.environ['THE_SOFTWARE_DIR'])
else:
    print("Either 'THE_SOFTWARE_DIR' or 'THE_SOFTWARE_SOURCE_DIR' must be set!")
    sys.exit(-1)

systemPlatform = platform.system()
if systemPlatform != "Linux" and systemPlatform != "Windows":
    print("Could not determine system platform")
    sys.exit(-1)

if "BUILDDIR_DN" in os.environ:
    builddir_dn = Path(os.environ['BUILDDIR_DN'])
else:
    builddir_dn = the_software_dir / r"LanguageBindings/Python/Output"
    if systemPlatform == "Linux":
        os.system("export BUILDDIR_DN={}".format(builddir_dn))
    elif systemPlatform == "Windows":
        os.system("set BUILDDIR_DN={}".format(builddir_dn))
version_file = the_software_dir / r"DriverBase/Include/softwareVersionInfo.h"

class SoftwareVersion(object):
    def __init__(self):
        class BuildNr(str):
            def returnNr(self, key): # key is a str like 'THE_SOFTWARE_MAJOR_VERSION'
                offset = self.find(key)
                if offset == -1:
                    return '?'
                offset += len(key)
                while not self[offset].isdigit():
                    offset += 1
                nr = self[offset]
                while True:
                    offset += 1
                    if not self[offset].isdigit():
                        return nr
                    nr += self[offset]

        version_all_h = open(version_file)
        version = BuildNr(version_all_h.read())
        version_all_h.close()
        self.v_major = version.returnNr('#define THE_SOFTWARE_MAJOR_VERSION ')
        self.v_minor = version.returnNr('#define THE_SOFTWARE_MINOR_VERSION ')
        self.v_point = version.returnNr('#define THE_SOFTWARE_RELEASE_VERSION ')
        self.v_build = version.returnNr('#define THE_SOFTWARE_BUILD_VERSION ')

    def full(self):
        return self.v_major + '.' + self.v_minor + '.' + self.v_point + '.' + self.v_build

software_version = SoftwareVersion()

if len(sys.argv) > 1:
    if sys.argv[1] == 'return_software_version':
        print('%s.%s.%s' % (software_version.v_major,
                            software_version.v_minor,
                            software_version.v_point))
        sys.exit()

if systemPlatform == "Linux":
    if( platform.architecture()[0] == "32bit" ):
        bits = 32
    elif( platform.architecture()[0] == "64bit" ):
        bits = 64
    else:
        print("Error! Could not determine the systems CPU architecture!")
        sys.exit(-1)
    machine = platform.machine()
elif systemPlatform == "Windows":
    bits = 64 if os.environ["PROCESSOR_ARCHITECTURE"] == "AMD64" else 32

macros = []
defines = os.getenv('DEFINES')
if defines is not None:
    for dfn in defines.split():
        macros += [(dfn[2:],'1')]

class dirlist(list):
    """You can define multiple directories in one environment variable"""
    def addDir(self, d):
        if type(d) is list:
            for a in d:
                self.addDir(Path(a)) # recurse
        else:
            d = Path(d)
            if os.pathsep in str(d): # turn 'bar;foo' into ['bar', 'foo']
                self.addDir(str(d).split(os.pathsep)) # recurse
            else:
                d = Path(str(d).replace('\ ', ' '))
                if d.is_dir():
                    self.append(d)
    def getList(self):
        return list(self)

inc_dirs = dirlist()
lib_dirs = dirlist()

# inc_dirs     : additional include directories
# lib_dirs     : additional library directories
# macros       : defined while building the extension

link_args = []
compile_args = []
# for debugging the Python extension library with Visual Studio on Windows set the variable in the next line to 'True'!
generateDebugInfo = False

installation_path = os.getenv('BUILD_FROM_INSTALLATION') # None if not set
if installation_path == '0': # '0' is explicit for 'not from installation'
    installation_path = None #  (like not specifying BUILD_FROM_INSTALLATION at all)
python_ver = '.'.join(str(x) for x in sys.version_info[:2])
if systemPlatform == "Linux":
    if "THE_SOFTWARE_SOURCE_DIR" in os.environ:
        if ("THE_SOFTWARE_BUILD_DIR" in os.environ):
            lib_dirs.addDir(Path(os.environ["THE_SOFTWARE_BUILD_DIR"]) / Path('lib'))
        else:
            print()
            print("ERROR: THE_SOFTWARE_SOURCE_DIR will be used but THE_SOFTWARE_BUILD_DIR is not set! The build directory thus cannot be found!")
            sys.exit(-1)
    else:
        if ((machine == "armv7l") or (machine == "armv7")):
            target_lib_sub_dir = Path('lib/armhf')
        elif (machine == "i686" or machine == "x86_64") :
            target_lib_sub_dir = Path('lib/x86_64')
        elif ( machine == "aarch64") :
            target_lib_sub_dir = Path('lib/arm64')
        else:
            print()
            print("ERROR: Platform unsupported!")
            sys.exit(-1)
        lib_dirs.addDir(the_software_dir / target_lib_sub_dir)
    compile_args = ['-Wno-unknown-pragmas', '-Wno-misleading-indentation', '-DCOMPILE_PYTHON']
    if ((machine == "armv7l") or (machine == "armv7")):
        compile_args += ['-O0']
        link_args = ['-O0']
elif systemPlatform == "Windows":
    compile_args = ['/GR','/bigobj', '/EHsc', '/DCOMPILE_PYTHON']
    if generateDebugInfo:
        compile_args += ['/Zi']
        link_args += ['/DEBUG']
    if "THE_SOFTWARE_SOURCE_DIR" in os.environ:
        if bits == 32:
            if ("THE_SOFTWARE_BUILD_X86_DIR" in os.environ) and ("THE_SOFTWARE_BUILD_CONFIGURATION" in os.environ):
                build_lib_dir = Path(os.environ["THE_SOFTWARE_BUILD_X86_DIR"]) / r"lib\win\win32" / os.environ["THE_SOFTWARE_BUILD_CONFIGURATION"]
            else:
                print()
                print("ERROR: THE_SOFTWARE_SOURCE_DIR will be used but THE_SOFTWARE_BUILD_X86_DIR is not set! The build directory thus cannot be found!")
                sys.exit(-1)
        else:
            if ("THE_SOFTWARE_BUILD_X64_DIR" in os.environ) and ("THE_SOFTWARE_BUILD_CONFIGURATION" in os.environ):
                build_lib_dir = Path(os.environ["THE_SOFTWARE_BUILD_X64_DIR"]) / r"lib\win\x64" / os.environ["THE_SOFTWARE_BUILD_CONFIGURATION"]
            else:
                print()
                print("ERROR: THE_SOFTWARE_SOURCE_DIR will be used but THE_SOFTWARE_BUILD_X64_DIR is not set! The build directory thus cannot be found!")
                sys.exit(-1)
        lib_dirs.addDir(build_lib_dir)
        link_args.append('/PDB:{}'.format(build_lib_dir / 'lib_the_software.pdb'))
    else:
        if bits == 32:
            target_lib_sub_dir = Path('lib')
        else:
            target_lib_sub_dir = Path(r'lib\win\x64')
        lib_dirs.addDir(the_software_dir / target_lib_sub_dir)

inc_dirs.addDir(the_software_dir)
inc_dirs.addDir(the_software_dir / 'theSofware_CPP')
if systemPlatform == "Windows":
    inc_dirs.addDir(the_software_dir / 'softwareComponent' / 'Include')

build_options = {'build_base' : builddir_dn}
bdist_options = {'bdist_base' : builddir_dn, 'dist_dir' : builddir_dn}

# Create an 'Extension' object that describes how to build the Python extension.
def createExtension():
    libs = ['softwareTool']

    incl_dirs = inc_dirs.getList()
    libr_dirs = lib_dirs.getList()

    global installation_path
    if installation_path is not None:
        installation_path = Path(installation_path)
        incl_dirs.insert(0, installation_path)
        incl_dirs.insert(0, installation_path / 'softwareComponent' / 'Include')
        incl_dirs.insert(0, installation_path / 'theSofware_CPP')
        libr_dirs.insert(0, installation_path / 'lib')
    else:
        incl_dirs.insert(0, the_software_dir)
    sources_fpns = [Path(builddir_dn) / 'software_wrap.cpp']
    return Extension('lib_the_software',
        sources       = list(map(str, sources_fpns)),
        include_dirs  = list(map(str, incl_dirs)),
        library_dirs  = list(map(str, libr_dirs)),
        define_macros = macros,
        libraries     = libs,
        extra_compile_args = compile_args,
        extra_link_args = link_args)

class build_ext_posix(build_ext):
    # on Linux, once the lib has been built, dump a map, then strip the lib
    def build_extensions(self):
        self.check_extensions_list(self.extensions)
        for ext in self.extensions:
            self.build_extension(ext)
            libName = str(Path(self.build_lib) / 'theSofware' / ext.name)
  
def getPlatform():
    platformList = []
    if systemPlatform == "Linux":
        platformList.insert(0, os.system('ldconfig -p | grep softwareTool | cut -d \> -f 2 | grep -o "/lib/.*/" | cut -d/ -f 3'))
    elif systemPlatform == "Windows":
        platformList.insert(0, os.environ["PROCESSOR_ARCHITECTURE"])
    return platformList

setup(
    name = "TheSoftware", # remove this once we require setuptools 61.0.0 or above
    version     = software_version.full(),
    platforms   = getPlatform(),
    description = 'The Software Python {} interface'.format(python_ver),
    ext_package = 'theSofware', # all modules are in 'theSofware'
    ext_modules = [createExtension()],
    package_dir = {'theSofware' : 'Output'},
    packages    = ['theSofware', 'theSofware.Common'],
    cmdclass    = {'build_ext' : build_ext_posix if os.name == 'posix' else build_ext},
    options     = {'build' : build_options, 'bdist' : bdist_options}
)

Output

$ python setup.py install
x86_64
x86_64
x86_64
running install
/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/_distutils/cmd.py:66: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  self.initialize_options()
running bdist_egg
running egg_info
creating ThePackage.egg-info
writing ThePackage.egg-info/PKG-INFO
writing dependency_links to ThePackage.egg-info/dependency_links.txt
writing top-level names to ThePackage.egg-info/top_level.txt
writing manifest file 'ThePackage.egg-info/SOURCES.txt'
reading manifest file 'ThePackage.egg-info/SOURCES.txt'
Traceback (most recent call last):
  File "/opt/ThePackage/LanguageBindings/Python/setup.py", line 240, in <module>
    setup(
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 184, in setup
    return run_commands(dist)
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
    dist.run_commands()
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/dist.py", line 950, in run_command
    super().run_command(command)
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/command/install.py", line 97, in run
    self.do_egg_install()
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/command/install.py", line 149, in do_egg_install
    self.run_command('bdist_egg')
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/dist.py", line 950, in run_command
    super().run_command(command)
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/command/bdist_egg.py", line 159, in run
    self.run_command("egg_info")
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/dist.py", line 950, in run_command
    super().run_command(command)
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 311, in run
    self.find_sources()
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 319, in find_sources
    mm.run()
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 545, in run
    self.prune_file_list()
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/command/sdist.py", line 161, in prune_file_list
    super().prune_file_list()
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 394, in prune_file_list
    self.filelist.exclude_pattern(None, prefix=build.build_base)
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/_distutils/filelist.py", line 249, in exclude_pattern
    pattern_re = translate_pattern(pattern, anchor, prefix, is_regex)
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/_distutils/filelist.py", line 357, in translate_pattern
    prefix_re = glob_to_re(prefix)
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/site-packages/setuptools/_distutils/filelist.py", line 318, in glob_to_re
    pattern_re = fnmatch.translate(pattern)
  File "/home/user/.pyenv/versions/3.9.18/lib/python3.9/fnmatch.py", line 89, in translate
    i, n = 0, len(pat)
TypeError: object of type 'PosixPath' has no len()

[abravalheri]

Hi @Mindstan, does the package install correctly if you use pip install --use-pep517 .?

setup.py is supported as a configuration file but no longer as a CLI tool.

 I can't share there the content of the setup.py.

Please note that without a reproducer, we cannot investigate or help you much. The good side is that to create a minimal reproducer, you don't need to disclose any proprietary information. It would be ideal if you can create some simplified toy example that triggers the same error, without disclosing the original source code.

Please have a look at the excellent guide from stack overflow on this topic: https://stackoverflow.com/help/minimal-reproducible-example.

If you are not able to provide a reproducer, maybe then the best thing is to contact the support from your supplier about this problem.

[Avasam]

@abravalheri A guess, but it looks like #4603 may have incidentally broken support for a build.build_base of type pathlib.Path by falling back to setuptools._distutils.command.sdist.sdist.prune_file_list, which uses self.filelist.exclude_pattern(None, prefix=build.build_base) instead of self.filelist.prune(build.build_base)

[Avasam]

Idk about OP's usecase, but here's a minimal reproducer that fails with the same error on v74 but passes on v73.0.1

from pathlib import Path

from setuptools import setup
from setuptools.command.build import build

class MyBuild(build):
    def initialize_options(self) -> None:
        super().initialize_options()
        self.build_base = Path.cwd()
        return

setup(cmdclass={"build": MyBuild})

python setup.py sdist
or
pip install .

[abravalheri]

Thank you @Avasam, that makes sense...

The thing is: I don't think that support for build.build_base of type pathlib.Path was intentional, or documented... Until very recently all parameters and arguments passed to setuptools representing paths were meant to be strings.

distutils started driving adoption/support for Path objects in pypa/distutils#272, but that does not include build_base.
The changelog only mention "for Pathlike objects in data files and extensions".

So that is probably another case of Hyrum's law.

We might have a look on this, but meanwhile users are can revert to strings to avoid problems.

@Mindstan, it is still important for you to provide a minimal reproducer of the original problem before we keep investigating to avoid relying on conjectures.

[Mindstan]

does the package install correctly if you use pip install --use-pep517 .?

I got the same error, with pip cli wrapping it.

I updated the initial post with the content of setup.py, I don't have time today to clean up to shrink the content to only the problematic code.

[jaraco]

@abravalheri A guess, but it looks like #4603 may have incidentally broken support for a build.build_base of type pathlib.Path by falling back to setuptools._distutils.command.sdist.sdist.prune_file_list, which uses self.filelist.exclude_pattern(None, prefix=build.build_base) instead of self.filelist.prune(build.build_base)

Nice analysis. I concur.

I don't think that support for build.build_base of type pathlib.Path was intentional, or documented... Until very recently all parameters and arguments passed to setuptools representing paths were meant to be strings.

Yes, indeed, but it still seems worthwhile to restore incidental support where it was present but lost.

[abravalheri]

Yes, indeed, but it still seems worthwhile to restore incidental support where it was present but lost.

Thank you @jaraco . I think we can add this in setuptools by overwriting the method, but I think it would also fit nicely in distutils (we probably just need to add os.fspath). Which approach would you like me to take?

[jaraco]

I'm happy to handle this one, because I'd like to see the approach in distutils.

[Mindstan]

I confirm I have no longer the issue with 74.1.2. Thanks!