Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build QEMU-emulated Linux aarch64 wheels on GitHub Actions instead of Travis CI #7690

Merged
merged 10 commits into from
Jan 4, 2024

Conversation

hugovk
Copy link
Member

@hugovk hugovk commented Jan 3, 2024

Fixes #7390.
Closes #7485.

Follow on from discussion starting #7485 (comment).

This PR replaces the native Linux aarch64 built on Travis CI, and instead builds them on GitHub Actions using QEMU emulation.

  • The native builds are fast: 3 jobs of 7+7+5 wheels = 19 wheels in 40m + 40m + 20m = 1h40m total build time, about 40 mins waiting time.

  • Emulated builds are much slower: one job per each of 19 wheels, taking between 1h25m and 2h10m. Waiting time about 2h10m.

Because they're slow, we build each wheel in its own job, to make full use of the 20 parallel jobs available.

The jobs are started in alphabetical order of job ID:

  • We want to start the slowest QEMU first (19 x 1h25m-2h10m) to give them a headstart.
  • Then Linux and macOS native wheels next (4 x 15m-40m).
  • Then Windows wheels (3 x 10-35m).
  • Finally the single sdist (30s).

Of the QEMU wheels, the CPython builds take around 1.5h and PyPy just over 2h. These are started in by order of the matrix variables (see https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs#using-a-matrix-strategy). This means the PyPy ones unfortunately come last.

We can work around this, like add another variable to the matrix to make PyPy come first. Or split the config like build-1-QEMU-emulated-wheels-pypy and build-2-QEMU-emulated-wheels-cpython. But I've left that for now -- with 19 QEMU jobs and a clear CI queue, they should all get started at the same time. Maybe something to consider later.

This will increase total time for the release workflow from around 50 minutes to maybe 2h15m, but we no longer rely on Travis CI and the whole build and deploy to PyPI takes place in a single workflow, to make releases easier, more reliable and secure.

@radarhere radarhere merged commit 4233dd7 into python-pillow:main Jan 4, 2024
85 checks passed
@hugovk hugovk deleted the qemu branch January 5, 2024 05:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Set up GitHub release token for Travis CI Automate build and release to PyPI
2 participants