universal2 wheels for macOS

[Pantsworth]

Hello! It would be really helpful to have universal2 builds of Pillow available for macOS in addition to the x86_64 and arm64 builds.

Having a single wheel that supports both platforms makes it a lot easier to release tools across both macOS architectures.

[radarhere]

If it is helpful to have an immediate solution, here are some universal2 builds for Python 3.10 and 3.11 -
universal2.zip

[Pantsworth]

Thanks @radarhere ! Great to have those.

[radarhere]

I've created multi-build/multibuild#495 to help with this.

Here are universal2 builds for Python 3.8 and 3.9 - universal2.zip

[hugovk]

Thank you for the PR!

Do we need universal2 in addition to x86_64 and arm64 builds?

I'm mindful we have about 60 wheels per release, and expect other architectures will be added at some point, using resources on PyPI (it costs a donated $1.8M per month to run PyPI!) and increasing the (2 hour+) build times here: ~15m each for x86_64 and arm64, and ~30m for universal2, multiplied for each Python version.

Looking at the advice at cibuildwheel (we use multibuild rather than cibuildwheel, but it's generic advice):

x86_64

The traditional wheel for Apple, loads on Intel machines, and on Apple Silicon when running Python under Rosetta 2 emulation.

Due to a change in naming, Pip 20.3+ (or an installer using packaging 20.5+) is required to install a binary wheel on macOS Big Sur.

macOS 11 Big Sur is the oldest supported, so this sets a baseline of pip 20.3 for replacing x86_64. But we require modern pip for installation anyway, and pip 20.3 is over two years old (Nov 2020).

arm64

The native wheel for macOS on Apple Silicon.

Requires Pip 20.3+ (or packaging 20.5+) to install.

Same pip version.

universal2

This wheel contains both architectures, causing it to be up to twice the size (data files do not get doubled, only compiled code). It requires Pip 20.3 (Packaging 20.6+) to load on Intel, and Pip 21.0.1 (Packaging 20.9+) to load on Apple Silicon.

Same pip version needed for replacing x86_64, slightly newer pip 21.0.1 (Jan 2021) for replacing arm64.

File sizes for each architecture are about the same, regardless of Python version:

5.8M    Pillow-9.4.0-cp38-cp38-macosx_10_10_universal2.whl
3.2M    Pillow-9.4.0-cp38-cp38-macosx_10_10_x86_64.whl
2.9M    Pillow-9.4.0-cp38-cp38-macosx_11_0_arm64.whl

5.8M    Pillow-9.4.0-cp39-cp39-macosx_10_10_universal2.whl
3.2M    Pillow-9.4.0-cp39-cp39-macosx_10_10_x86_64.whl
2.9M    Pillow-9.4.0-cp39-cp39-macosx_11_0_arm64.whl

5.8M    Pillow-9.4.0-cp310-cp310-macosx_10_10_universal2.whl
3.2M    Pillow-9.4.0-cp310-cp310-macosx_10_10_x86_64.whl
2.9M    Pillow-9.4.0-cp310-cp310-macosx_11_0_arm64.whl

5.8M    Pillow-9.4.0-cp311-cp311-macosx_10_10_universal2.whl
3.2M    Pillow-9.4.0-cp311-cp311-macosx_10_10_x86_64.whl
2.9M    Pillow-9.4.0-cp311-cp311-macosx_11_0_arm64.whl

universal2 (5.8M) is a bit under the combined size of the other two (6.1M).

I don't think 5.8M is too big for a single combined installer?

Note

The dual-architecture universal2 has a few benefits, but a key benefit to a universal wheel is that a user can bundle these wheels into an application and ship a single binary.

However, if you have a large library, then you might prefer to ship the two single-arch wheels instead - x86_64 and arm64. In rare cases, you might want to build all three, but in that case, pip will not download the universal wheels, because it prefers the most specific wheel available.

So they suggest there's limited benefit building all three.

Generally speaking, because Pip 20.3 is required for the universal2 wheel, most packages should provide both x86_64 and one of universal2/arm64 wheels for now. When Pip 20.3+ is common on macOS, then it might be possible to ship only the universal2 wheel.

This advice is from Feb 2021 (pypa/cibuildwheel#577), two years later we should be fine with only universal2?

Or if we want to be cautious, how about x86_64 and universal2 now for the next 9.5.0 release (April 2023)?

And because Python 3.7 can only use x86_64, and we're dropping 3.7 in the following 10.0.0 release (July 2023), we could at the same time switch to universal2 only? That would simplify config too. (We'd still need x86_64 for PyPy.)

[radarhere]

A possible alternative would just be to document how users can create universal2 wheels themselves from our existing x86_64 and arm64 wheels.

$ python3 -m pip download --only-binary=:all: --platform macosx_10_10_x86_64 Pillow
Collecting Pillow
  Using cached Pillow-9.4.0-2-cp39-cp39-macosx_10_10_x86_64.whl (3.3 MB)
Saved ./Pillow-9.4.0-2-cp39-cp39-macosx_10_10_x86_64.whl
Successfully downloaded Pillow
$ python3 -m pip download --only-binary=:all: --platform macosx_11_0_arm64 Pillow
Collecting Pillow
  Using cached Pillow-9.4.0-cp39-cp39-macosx_11_0_arm64.whl (3.0 MB)
Saved ./Pillow-9.4.0-cp39-cp39-macosx_11_0_arm64.whl
Successfully downloaded Pillow
$ python3 -m pip install delocate

from delocate.fuse import fuse_wheels
fuse_wheels('Pillow-9.4.0-2-cp39-cp39-macosx_10_10_x86_64.whl', 'Pillow-9.4.0-cp39-cp39-macosx_11_0_arm64.whl', 'Pillow-9.4.0-cp39-cp39-macosx_11_0_universal2.whl')

I've created python-pillow/Pillow#6912 for this.

[radarhere]

Generally speaking, because Pip 20.3 is required for the universal2 wheel, most packages should provide both x86_64 and one of universal2/arm64 wheels for now. When Pip 20.3+ is common on macOS, then it might be possible to ship only the universal2 wheel.

This advice is from Feb 2021 (pypa/cibuildwheel#577), two years later we should be fine with only universal2?

It might be worth noting that before 20.3, pip only supported Python < 3.9 - https://pypi.org/project/pip/20.2.4/. So at the very least, when we drop support for Python 3.8 with Pillow 11, the end user will have to be using pip 20.3.

[radarhere]

python-pillow/Pillow#6912 was merged, and the documentation can now be seen under the macOS tab at https://pillow.readthedocs.io/en/stable/installation.html#basic-installation

[Pantsworth]

That's super helpful, and I think will work just as well for our use case. Thanks!

[hugovk]

Given #359 (comment), and we've documented how to create a combined universal2 wheel (python-pillow/Pillow#6912), I don't see a need to also produce universal2 wheels in addition to the single-architecture ones.