Check if using xdist may help with performance problems

[Czaki]

This PR is to check how usage of pytest-xdist change time of test execution

[Czaki]

for linux this double speed of test.

[Czaki]

@joerick @YannickJadoul There is any reason to use symlinks on macos? Or it could be simply switched to PATH like in other systems?

[joerick]

The symlinks were required because we wanted python to always be the active Python, but in some versions it was only available as python3. But, the SYMLINKS_DIR could be a temporary directory, instead of a hardcoded path.

[YannickJadoul]

Ah, yes. And I believe mostly @joerick preferred to keep the installation directory itself clean, rather than adding extra symlinks in there (which, yes, why not; it keeps things clean).

[Czaki]

@joerick @YannickJadoul As see performance problem in #430 I return to this PR.
This what I found Is that I need to use separate python interpreters group for each worker (If run 2 workers, then install python interpreters twice. (this problem not happens for linux).

I see how do this for windows (thanks for nuget). But there is problem that utils.cibuildwheel_run use subprocess.run

What do you think about switch to direct python call using mocking? Or have other idea?

I do not see how to deal with macos. I do not see nuget like python installer for macos.

This PR add complexity to test process but could speedup test run. What do you think? Push it?

[YannickJadoul]

As @joerick pointed out, the problems in #430 seem to be pointing at some other underlying issue. So while this can be useful in general, it's definitely not a good idea to use it right now as workaround, before we figured out what's wrong or why #430 is so much slower.

Apart from that, I wonder how confusing it's going to be to have tests run in parallel. At any rate, I'd say to just run a single job (test_0_basic) beforehand, such that everything necessary is installed, rather than patching in other ways.

Also, maybe it's better to make the separate pytest runs parallel, rather than inside each of these runs? But again, I'm not sure.

[Czaki]

I start this PR to reduce time of test execution. reduce time from 34 minutes to 22 minutes (linux case) is really nice.
The #430 was reason to come back to this PR. Do not treat this as workaround for #430 problem. Expected speedup factor is bellow 2.

The problem is that interpreter does not allow pararell call of pip wheel.

Other option which I see Is to parallel loop inside build functions.

[YannickJadoul]

Other option which I see Is to parallel loop inside build functions.

No, we don't want that, I think. That makes all execution of cibuildwheel parallel; I really don't think that's a good idea.

[Czaki]

No, we don't want that, I think. That makes all execution of cibuildwheel parallel; I really don't think that's a good idea.

I do not understand. When using ThreadPoolExecutor from concurrent.futures you could set number of pararell workers.

When give up making symlinks on macos it should work. And Each worker work with other python version.

[YannickJadoul]

I do not understand. When using ThreadPoolExecutor from concurrent.futures you could set number of pararell workers.

No, but building wheels is complicated enough already that we don't want to introduce any concurrency within cibuildwheel, I'd say. If users want to build parallel, they can set flags in setup.py for parallel compilation to speed up building wheels (I'm doing that), but I think cibuildwheel should stay as simple and transparent as possible here. Running our own CI in parallel, maybe; but introducing concurrency to the build function, I really don't like that.

[joerick]

Closing to tidy things up. Feel free to reopen if you're still working on it!

[YannickJadoul]

@Czaki, @henryiii recently made some relevant PRs in pybind11: pybind/pybind11#2521

[Czaki]

@Czaki, @henryiii recently made some relevant PRs in pybind11: pybind/pybind11#2521

The main problem in this PR is parallel calls of pip wheel and changes in the environment (like install packages) I think that I will try to go back to this or when pip whell will be replaced with a new building tool.

Pip does not allow on pararell call.

[YannickJadoul]

That's exactly why I'm pointing to a way to make setuptools itself run in parallel, rather than running multiple builds in parallel (which to me is a lot more tricky).

[Czaki]

That's exactly why I'm pointing to a way to make setuptools itself run in parallel, rather than running multiple builds in parallel (which to me is a lot more tricky).

But in each project, we compile a single file. Also, compilation time is not the longest part of our pipeline, but coping files, creating environments (each call of pip wheel will create a new virtualenv) etc.

[YannickJadoul]

Yes, but aren't our tests currently OK? I thought you wanted to introduce this in cibuildwheel in general, for bigger projects.

[YannickJadoul]

Nevermind. I see I'm completely utterly wrong. Sorry for the confusion!

[Czaki]

As a conclusion of the current state.
xdist works fine with Linux, because of using docker images.
It may be used for windows id create two sets of python installation (one for each test worker).

But does not work with macOS because it does not allow to install of two instances of python interpreter.