Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-108388: regrtest runs slowest tests first #108391

Closed
wants to merge 3 commits into from

Conversation

vstinner
Copy link
Member

@vstinner vstinner commented Aug 24, 2023

The Python test suite (regrtest) now runs the 20 slowest tests first and
then other tests, to better use all available CPUs when running tests in
parallel.

XXX: for tests, the PR also reverts the commit 7a6cc3e.

The Python test suite (regrtest) now runs the 20 slowest tests first and
then other tests, to better use all available CPUs when running tests in
parallel.
@gpshead
Copy link
Member

gpshead commented Aug 24, 2023

I've thought about doing this in the past. It sort of defeats the purpose of the regrtest -r randomization. It also means people running the testsuite themselves don't get a nice feeling of general feedback on their change immediately because the test run just sits there using all 20 of their cores on long tests for a 30-60s before anything is available to run our 400 fast tests that collectively otherwise complete within the first 20 seconds.

Perhaps have this look at the -j number and only schedule the 0.7*j slowest tests up front.

On Github actions CI systems with basically only one core (two hyperthreads is basically 1.1-1.2 cores), the ordering of fast vs slow tests is not likely to make much difference.

buildbots sometimes have more cores though.

@vstinner
Copy link
Member Author

It we manage to split enough tests so no test takes longer than 1 minute, it should help a lot of the OS scheduler to better use all available CPUs, and so this change becomes less relevant, or even useless.

I close this PR for that. I wanted to give it a try to see how it looks.

But when I used it, it was bad that no test completed in the first 30 seconds, as if something was broken :-( So yeah, for users, it's unpleasant :-( We should investigate other approaches to make tests faster and keep them "responsible" (display progress to users ;-)).

@vstinner vstinner closed this Aug 24, 2023
@vstinner vstinner deleted the regrtest_slowest_first branch August 24, 2023 16:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants