Solving issues related to out-of-tree builds

[sbidoul]

I open this issue as an attempt to consolidate discussion about out-of-tree builds, the related issues and possible solutions.

What's the problem this feature will solve?

When building projects from local directories, pip first copies them to a temporary location.

This approach has raised a number of issues over time:

• When setup.py / pyproject.toml is not at the root of the project and when the build depends on resources outside of the setup.py / pyproject.toml subtree (pip wheel . does not follow symlinks whereas python setup.py bdist_wheel does #3500, Modify pip build behavior to better support tools like setuptools_scm #7549, PEP 517 isolation breaks test module with relative out-of-source dependency #6276), e.g.:
  • build needs resources that are symlinks to files/directories outside of the subtree
  • build needs the git repository (eg when using setuptools_scm), and .git/ is in a parent directory not copied to the temp dir by pip
  • build relies on the subdirectory name (somewhat exotic maybe, yet I have such as case where I want to create a custom build backend and part of the metadata depends on the subdirectory name)
• Performance issues when the the project directory is large (pip install of a directory is super slow #2195).

Why does pip copy to a temporary directory before building? Caveat: this is unclear to me - here is what I collected so far:

• To avoid relying to something out of source (pip install of a directory is super slow #2195 (comment)) - although the definition of "out of source" is the cause of some issues above
• To avoid polluting the source directory with build artifacts or residues (?)
• Something else?

Possible Solutions

1. Build an sdist in place, unpack the sdist in a temporary location, then build from that.
2. Add a pip option to build in place.
3. Update PEP 517 with some sort of mechanism to let back-ends communicate to front-ends if they are "safe" for in-place builds.
4. Change pip to always build in place.
5. Change pip to build in place by default with an option to build out-of-tree.

Additional context

More discussion about building via sdist on discuss.python.org.

[pfmoore]

Looking at this from the back-end's perspective, the idea of an "out of tree build" is actually meaningless. The back end is given a "source tree" in the form of the current directory of the process, and is asked to execute a build. It's got no way of knowing whether that directory was extracted from a sdist, checked out of a VCS, or copied from somewhere else. All it can do is build, and if it doesn't have what it needs to build, report failure.

So that pretty much kills possible solution (3) IMO - the backend doesn't have a concept of what an in-place build is, so it can't say whether such a build is safe¹.

As regards (2), I'm generally opposed to additional options like this. If we know what the best thing to do is, we should do it, and if we don't, then passing the problem onto the user is not a particularly friendly option. The issue here is subtle enough that I would expect few users to know what the correct choice is, so we'd likely see people just trying the 2 options and blindly using "whatever works". Also, the support implications are significant - offering an option clearly implies that we expect the build results to be different in at least some cases, so how do we test that the differences are as we'd expect? Are we responsible for educating users on when the in-place build flag would be needed or not (either via our documentation, or as a result of issues being raised by users who don't know which to use)?

Having said that, I'm not against pip simply building in place. I believe the reason we don't is because we had cases where artifacts left over from a previous build were used in a subsequent build, but were built with different options (for example, object files built in debug mode being linked into a release build done from the same tree). It is, however, reasonable to say that backends should ensure that issues like this don't happen, and if they do, it's a backend bug and not up to pip to try to defend against it. However, I'm not sure if taking such a stance is particularly user friendly - this came up during the PEP 517 discussions under the topic of incremental builds, and was controversial enough that it was deferred at the time.

My preference has been to build a sdist, and then build the wheel from that, for a long time now (your option 1). I'm OK with pip building in place (if we can agree that doing so is safe, given that as I noted above, we can't expect the back-end to let us know), but I think that would need a community discussion to thrash out the wider implications (on backends, pip/frontends and end users).

¹ It would of course be possible to update PEP 517 to define what constitutes an "in-place" source tree as opposed to an "out of tree" build, but I suspect that would be a very difficult concept to pin down.

[sbidoul]

I added solutions 4. (Change pip to always build in place) and 5. (Change pip to build in place by default with an option to build out-of-tree).

[sbidoul]

I've mixed feelings regarding building via sdist (solution 1.) for the following reasons:

1. some projects may have a broken sdist-to-wheel path; while I see the value of validating that building from sdists works, doing it by default now will certainly break a lot of end-user builds
2. it still has performance implications for projects with large sdists, and due to additional subprocess calls
3. I personally think we should not put too much emphasis on sdists as it is quite common for project to publish built artifacts and refer to their favorite code hosting platform for their source releases (for instance, backends that rely on the presence of a VCS checkout for building need to jump through hoops to produce a working sdist). And PEP 517 specifically allows for backends to raise UnsupportedOperation for build_sdist.

This post on discuss also summarizes similar arguments.

I agree the path for solution 3. is far from obvious.

I also agree we should avoid additional options if we can.

I also note there were some voices in favor of in-place builds in the linked discuss thread, but indeed a focused community discussion on that specific subject is necessary if we want to explore that approach.

[pradyunsg]

• Build an sdist in place, unpack the sdist in a temporary location, then build from that.

I think this is a good approach.

IMO chances are, this would be run for a single package in most pip install runs involving local directories -- most commonly I imagine pip install .. This would be done likely as part of the package development workflow.

Here's what I think this behavior should be:

• If the backend is unable to create an sdist, do local-dir -> wheel (in-place)
  • I think the onus is very much on the backend to make sure that local-dir -> wheel is an idempotent operation in this case.
• If the backend is capable of creating an sdist, do local-dir -> sdist -> unpacked-sdist -> wheel.

Doing local-dir -> sdist -> wheel, we do have an additional set of calls. However, I do think it's reasonable to to validate that generated sdists are sane, especially during development. tox already does this as part of its workflow, check-manifest to cover for setuptools' not-so-friendly interfaces here.

Overall, I do think the costs of building an sdist when given a local directory are worth it to prevent such errors in projects, especially since folks installing from local directories are likely the developers of the project themselves.

Regarding the rollout, I think we'd want to wait and watch how #6536 comes around. We'd definitely learn a few things that can be transferred over.

[chrahunt]

I prefer building/installing in-place without sdist (so setup.py install or setup.py bdist_wheel or calling build_wheel on the PEP 517 backend as applicable) vs building an sdist, unpacking, and installing from that. My specific reasons are:

1. Supports the largest number of users out of the box.
   1. Assume pip does local-dir -> installed (via wheel build in-place or setup.py install). Users that want to go from local-dir -> installed can execute pip install local-dir. Users wanting to go from local-dir -> sdist -> installed are free to create an sdist and then execute pip install ./path/to/sdist. There are plenty of alternate tools that can build an sdist, and this is something that users will be more likely to have already unless they are hand-crafting the distributions they upload to PyPI.
   2. Now assume pip does local-dir -> sdist -> installed. Users that want to go from local-dir -> installed have no options that involve pip. Users wanting to go from local-dir -> sdist -> installed can execute pip install local-dir. The users with no options will ask pip for options to control the behavior or will have to find another tool that they wouldn't've otherwise needed.
2. If we implement local-dir -> sdist -> installed, presumably we would also do it for VCS-based requirements? If so that's more work. If not, that's additional paths through the code and deviances in the way installation is handled that have to be remembered by users or when providing support.
3. Least amount of work to implement. There are three places to change to implement local-dir -> installed (here, here, and here). For local-dir -> sdist -> installed I wouldn't even want to touch implementing this until Build Logic Refactor #6607 is done, otherwise I think it would end up in a lot of places in the code base similar to artifact downloading/hash checking.
4. Least amount of work to test since, IMO, existing tests are sufficient to cover the local-dir -> installed code path. For the local-dir -> sdist -> installed path we would want to verify that we're actually building an sdist, and that the fallbacks work for building a wheel directly.
5. Least amount of work (computationally). As mentioned elsewhere, local-dir -> sdist -> installed is an additional sub-process call (and that sub-process is doing work). It also means pip has to unpack the sdist (hello virus scanners and otherwise slow disks) before doing the wheel build.

Regardless of approach, the only issue I see with doing things in-place is that for setuptools builds (I think this applies for legacy and PEP 517) we would end up with .egg-info in the project directory which will get mistaken as an "installed package" when pip is invoked with python -m pip in that directory. This would be fixed by #4575 which would presumably NOT include the current directory in the query for installed packages for any scheme.

[pradyunsg]

Noting that I've grown to agree that the idea of skipping the sdist build and directly doing an in-tree build is a better approach for pip to take by default, and not trying to do local-dir -> sdist -> wheel.

[hroncok]

In Fedora, when we build Python RPM packages, we are dinosaurs and the standard way to do thins is to use python setup.py build. With PEP 517 we added a "provisional" way of using pip wheel instead. However with extension modules we have a problem with the "move sources to tmp, build from there" approach that pip uses to build them.

Our build machinery is injecting some compiler flags so the build artifacts (.so extension modules in this case) contain metadata about their sources. Later there is a shell script that traverses the build artifacts, extracts this information and copies the sources to /usr/src/debug to be installed via a special *-debugsource RPM. The mahcinery expects everything to be built within the working tree and it doesn't really work nice when built outside. Here are the things that can be done (together) to mitigate the problem on our side:

1. set the $TMPDIR environment variable to have it within the place where the RPM script expects it (i.e. export TMPDIR=%{_builddir}/.tmp (and create it))
2. use pip wheel with the --no-clean option to keep the copied sources in $TMPDIR
3. run some shell kung fu to rewrite the "what is my source" information to the correct location: find %{buildroot} -iname '*.so' -print0 | xargs --no-run-if-empty -0 -n1 /usr/lib/rpm/debugedit -b "%{_builddir}/.tmp/pip-req-build-"* -d "$PWD"
4. (Optional: clean $TMPDIR manually.)

We don't particularly like the third step because it relies on too many implementation details:

• /usr/lib/rpm/debugedit API and location (and existence)
• the pip-req-build name
• the mechanism with which pip builds the sources

If pip would always build in place or if there would be command line switch for this, the issue would go away.

Downstream report: https://bugzilla.redhat.com/show_bug.cgi?id=1806625

[pfmoore]

Noting that I've grown to agree that the idea of skipping the sdist build and directly doing an in-tree build is a better approach for pip to take by default, and not trying to do local-dir -> sdist -> wheel.

I'm also becoming more inclined to accept the idea of doing an in-place wheel build. My remaining reservations are:

1. We'd be relying on the backend behaving "correctly" - e.g., not giving different results based on left-over data from previous builds, or whatever. I'm fine with making that assumption, but I'm concerned about the support cost if we start getting people saying "pip built my wheel wrongly" and we have to debug only to find that it's a backend issue.
2. I think that whatever approach we take, we should very clearly and explicitly document it, and make a strong effort to move completely over to it. We really don't want to add "yet another approach" to pip while leaving all of the old code around for compatibility. If we can't commit to the new approach, we're just introducing yet more maintenance headaches.
3. As a corollary to the above, we should make sure there are no projects that we know of which rely on our current "copy and build" approach, as if there are we'll be breaking them with this change.

... and of course someone will need to write a PR implementing this change (with tests, docs, etc - the usual stuff) otherwise all we're doing is talking 🙂

[uranusjr]

We'd be relying on the backend behaving "correctly" - e.g., not giving different results based on left-over data from previous builds, or whatever. I'm fine with making that assumption, but I'm concerned about the support cost if we start getting people saying "pip built my wheel wrongly" and we have to debug only to find that it's a backend issue.

Would it make sense to extend the PEP 517 interface to include a “clean” hook? We’d probably want it as some point anyway to enable other endeavours (e.g. implement editable install, building a package development tool that builds any PEP 517 project). pip can call it here to make sure there are no leftover junk before doing in-tree build.

[pfmoore]

Would it make sense to extend the PEP 517 interface to include a “clean” hook?

Maybe? But if pip automatically calls clean, there's bound to be someone who wants to not do so, to do incremental builds or something. And then we end up with another option.

My inclination is to stick with my position "we need to be able to assume that it's the responsibility of the backend to ensure that in-place builds work correctly". Even if that turns out to be untenable, getting concrete examples of why it doesn't work will help us better understand what to do about the issue, rather than just guessing.

[gaborbernat]

1. We'd be relying on the backend behaving "correctly" - e.g., not giving different results based on left-over data from previous builds, or whatever.

I'd be tempted to extend PEP-517 to make this an explicit requirement.

[pfmoore]

I'd be tempted to extend PEP-517 to make this an explicit requirement.

It already says this:

The backend may store intermediate artifacts in cache locations or temporary directories. The presence or absence of any caches should not make a material difference to the final result of the build.

It's not so much that backends are likely to violate this requirement deliberately, as that users will naturally report the issue as a pip issue, and get redirected to the backend project, which is a bit of extra overhead.

[hroncok]

When trying t figure a workaround for Fedora, we've been hit by #7872

[sbidoul]

Since we have resolved the ".egg-info in cwd" issue with #7731 and friends, that is one less thing to worry about when building in place.

[sbidoul]

So option 4 (always build in place) was implemented in #7882 .

[pradyunsg]

We have now (per #7951) published a beta release of pip, pip 20.1b1. This release includes #7882, which implemented a solution for this issue.

I hope participants in this issue will help us by testing the beta and checking for new bugs. We'd like to identify and iron out any potential issues before the main 20.1 release on Tuesday.

I also welcome positive feedback along the lines of "yay, it works better now!" as well, since the issue tracker is usually full of "issues". :)

[hroncok]

We totally plan to test it out in Fedora (we already planned that before your comment), however the Tuesday deadline is probably not realistic.

[pradyunsg]

@hroncok Any idea by when Fedora might be able to test out these changes?

[hroncok]

I'll try our best to do it somehow on Monday, however I cannot make any promises.

[hroncok]

Indeed, 20.1b1 makes our problems go away.

More general 20.1b1 feedback in https://mail.python.org/archives/list/python-dev@python.org/message/5EAUIYYIRKXEHTAG5GQ7EJHSXGZIW2F7/