Support cross compiling

[isuruf]

Even though meson supports cross compiling, it seems meson-python does not.

meson-python/mesonpy/__init__.py

Lines 313 to 348 in 78861f5

	@cached_property
	def _stable_abi(self) -> Optional[str]:
	"""Determine stabe ABI compatibility.
	Examine all files installed in {platlib} that look like
	extension modules (extension .pyd on Windows, .dll on Cygwin,
	and .so on other platforms) and, if they all share the same
	PEP 3149 filename stable ABI tag, return it.
	All files that look like extension modules are verified to
	have a file name compatibel with what is expected by the
	Python interpreter. An exception is raised otherwise.
	Other files are ignored.
	"""
	soext = sorted(_EXTENSION_SUFFIXES, key=len)[0]
	abis = []
	for path, _ in self._wheel_files['platlib']:
	if path.suffix == soext:
	match = re.match(r'^[^.]+(.*)$', path.name)
	assert match is not None
	suffix = match.group(1)
	if suffix not in _EXTENSION_SUFFIXES:
	raise ValueError(
	f'Extension module {str(path)!r} not compatible with Python interpreter. '
	f'Filename suffix {suffix!r} not in {set(_EXTENSION_SUFFIXES)}.')
	match = _EXTENSION_SUFFIX_REGEX.match(suffix)
	assert match is not None
	abis.append(match.group('abi'))
	stable = [x for x in abis if x and re.match(r'abi\d+', x)]
	if len(stable) > 0 and len(stable) == len(abis) and all(x == stable[0] for x in stable[1:]):
	return stable[0]
	return None

assumes that the extension is run on the same interpreter.
We use crossenv to cross compile in conda-forge and crossenv monkey-patches some things but monkey-patching importlib.machinery.EXTENSION_SUFFIXES does not seem like a good idea.

cc @h-vetinari, @rgommers, @eli-schwartz

[rgommers]

Thanks for identifying that issue @isuruf. It looks like we need to avoid this extension check, and also add a basic cross-compile CI job. I was just looking at that for SciPy. I'm not that familiar with crossenv, but I think that for meson-python CI we need a cross-compilation toolchain (maybe from dockcross) and also both host and build Python already installed - crossenv doesn't do that for you if I understood correctly. So we need a Docker image with those things - perhaps from conda-forge?

[dnicolodi]

I don't think calling this a bug is fair. Cross compilation is not supported by the Python packaging ecosystem. And apparently "crossenv monkey patches some things" to make it work, whatever this means, given that PEP 517 build are usually done out-of-process. The detection of the stable ABI is just one of the places where we introspect the running environment, and Meson does too. Python does not offer a supported way to obtain the information required to build an extension module for another interpreter, let alone for an interpreter running on another platform. If you need support for cross compilation, please bring the issue to the attention of the CPython developers.

[rgommers]

Fair enough, Python's support is very poor. But right now moving from setuptools to meson-python is a regression, because crossenv does this ad-hoc patching of just the things that setuptools needs. So it is important and we need to fix it. Ideally both with and without crossenv.

We're not that far off, I'm just messing around with trying to get a Linux x86-64 to aarch64 cross compile to work for SciPy with:

dockcross-linux-arm64 bash -c "python -m pip install . --config-settings=setup-args=--cross-file=/work/cross_aarch64.ini"

I suspect it only requires a few tweaks.

[dnicolodi]

The first obvious issue I can think about is that we use sys.maxsize to determine if the target Python interpreter is 32 or 64 bits. How do you plan to make that work? AFAIK setuptools uses the same check, thus setuptools does not support cross compiling between 32 and 64 bit architectures.

[eli-schwartz]

The detection of the stable ABI is just one of the places where we introspect the running environment, and Meson does too.

The difference is that meson specifically executes an external python installation with a json dumper script, in order to scrape for information which meson knows it may need. It looks like meson-python checks for this information in-process instead, which means that it is bound to the same version of python that it is running inside.

Cross compilation has two challenges:

• compile for something other than what you are
• compile for something you cannot exec on your hardware/software stack

[dnicolodi]

Well, meson-python implements PEP 517, which does not have any provision for cross-compiling. It assumes that the Python interpreter used for the build, is the one you are building for. More generally, there is no way to generate wheel tags for an interpreter that you cannot execute. And if you can execute the Python you are building for, why not use it to run meson-python? I know all this is not ideal. But supporting these interfaces with these constraints is what meson-python is set up to do. If we want to build a tool for cross compiling Python wheels, it would have to look very different.

[eli-schwartz]

Sure, I do understand and empathize with those issues. There is finally some interest in getting this to work once and for all, though, I think. :)

And if you can execute the Python you are building for, why not use it to run meson-python?

FWIW this is actually a complex topic. Particularly worthwhile to note:

• qemu makes it possible to more or less fully generically emulate other machines, with the restriction that this is Linux-specific
• WINE on Linux can emulate Windows

But actually doing so is slow. So you actually don't want to do this, at least not more than you can get away with. So, there's a benefit to emulating just the json dumper and then running the rest of the build natively.

[dnicolodi]

So, there's a benefit to emulating just the json dumper and then running the rest of the build natively.

I completely agree, but while it makes sense for Meson to work this way, I think it would be overkill for meson-python. But, because PEP 517, we don't even have to think about it: the interfaces we need to implement do not support this.

[rgommers]

There's several levels of making things work here:

1. Make it work with crossenv, with the same level of support as setuptools has
2. Make it work by figuring out what we actually need to know (wheel tags, `sys.maxsize & co) and then allowing a user to put that into a cross file
3. Make it work out of the box without the user having to do (2) (this one requires stdlib support)

(1) and (2) should be feasible in a shorter time frame. (3) is going to be a lot more painful.

[h-vetinari]

If you need support for cross compilation, please bring the issue to the attention of the CPython developers.

By necessity, conda-forge has built a lot of its packaging around cross-compilation (i.e. there aren't native osx-arm64 CI agents, so we need to cross-compile from osx-64). These builds might even flow back to the non-conda world occasionally.

So it's a classic case of Hyrum's law, where a lot of things have grown up around an implicit interface (in this case the possibility to monkey-patch in cross-compilation), that we cannot switch the conda-forge builds for scipy to meson unless we solve the cross-compilation part somehow.

I don't mind carrying patches or workarounds for a while (i.e. it doesn't need to have a sparkling UX), but it would be very beneficial to have it be possible at all.

[dnicolodi]

Cross compiling for an arm64 target on a x86 build machine on macOS is already supported with the same interface used by setuptools.

For other user cases, I'm afraid that making the monkey-patches required to make cross compilation work with setuptools also work for meson-python is not possible, unless someone defines somewhere the interfaces that we can use and the ones we cannot. Even then, the thing would be extremely fragile without a clear definition of how the interface that we can use work when cross-compiling.

importlib.machinery.EXTENSION_SUFFIXES here is only as a safety check. We can remove it. But I would be very surprised if there are no other things that break.

[rgommers]

Even then, the thing would be extremely fragile without a clear definition of how the interface that we can use work when cross-compiling.

This is true, but the good thing is that the build envs are much better under control, and regressions are not as painful. We're looking to support distros here, not make pip install mypkg --cross from source work for end users.

[eli-schwartz]

It saddens me a bit that people seem to think crossenv is the definition of how to cross compile. There are people who have been cross compiling python modules without knowing that crossenv exists (or I think in one case, being vehemently convinced that crossenv is horrible and the worst thing possible for the cross compilation community 🤷‍♂️).

I think the reality is some combination of "a number of different groups have independently discovered some key aspects, and have different ideas how to do the rest".

• There's no such thing as a crossenv interface
• nor a cross-compile setuptools interface regardless of framework used to run setuptools

Meson has a cross-compile interface. Meson defines how to cross compile a meson project.

Frameworks used for cross compiling, including but not limited to crossenv, yocto, buildroot, voidlinux, etc, are responsible for interacting with the Meson cross-compile interface, and that is all. Meson, in turn, considers its cross compile interface to be "run a python interpreter and dump a specific list of values" -- this isn't well documented in the manual, but it does exist.

(Those projects may have also homebrewed their own cross compile interface for setuptools, but that really doesn't matter for either meson-python or for meson. At the end of the day, the tricks they use are irrelevant to meson except for the sysconfig tweaking, and that's just parallel evolution, not something owned by a specific tool.)

IMHO meson-python shouldn't fail to package a project that meson has successfully cross compiled, and for that reason I'm happy to see the check being removed. :)

If you want to validate that the ext suffix matches the binary architecture, that seems like a job for auditwheel or something.

If you need to generate a wheel tag, I actually feel a bit like that doesn't much matter. If you're building for local use you just immediately extract the results and discard the wheel tag in the process (and I assume conda has the same basic rationale to not-care) and again, I thought this was what auditwheel is for. If it can fix the manylinux version, I'd assume it can also fix a broken CPU architecture... can we just fall back on a nonsense wheel tag like "unknown"? Or lie and call it "none-any"?

[rgommers]

It saddens me a bit that people seem to think crossenv is the definition of how to cross compile.

I don't think that. crossenv is indeed just one of the ways, and seems to be a pragmatic hack to work around Python lack of support.

If it can fix the manylinux version, I'd assume it can also fix a broken CPU architecture... can we just fall back on a nonsense wheel tag like "unknown"? Or lie and call it "none-any"?

I agree with most of what you wrote, but not this. auditwheel is specific to manylinux-compliant wheels, and manylinux is not appropriate or needed in a number of circumstances. We do need to generate a correct wheel tag. It shouldn't be that hard.

It seems to me like meson-python does need to know that we're cross compiling. Detecting whether --cross-file is present in config_settings is probably reliable (covers everything except for the macOS environment variable way)? If so, could we just read the host machine section of the cross file, and generate the wheel tag from that?

[dnicolodi]

@eli-schwartz I agree with you on all points, except one: the tools that takes a Meson build directory and packs it up in a wheel is not meson-python, but something else, that may or may not share some code and be implemented in the same project as the current meson-python.

meson-python define itself has an implementation of PEP 517 https://peps.python.org/pep-0517/ In the interfaces defined in PEP 517 there is nothing that allows cross-compilation: it implicitly assumes that the wheels are being built for the Python interpreter running the build process. This is one of the reasons why solutions for cross compiling wheels have the taste of hacks: they need to work-around this interface limitation. AFAICT, auditwheel has the same core assumption, thus I don't think it can fix wheels built for an architecture that is not the one where it is being run.

Building on @eli-schwartz consideration that the correct cross-compilation interface is the one provided by Meson, we need a tools that allows access to that interface (forgetting PEP 517). However, wrapping Meson in another tool dedicated to build Python wheels is not necessary, what the tool needs is just the Meson build directory. I'm thinking to something that could be run like this:

meson setup $build_dir --cross-file $crossbuild_definition
meson compile -C $build_dir
meson-wheel-it-up $build_dir --tag $wheel_tag

Where meson-wheel-it-up is just an implementation of meson install that packs the build artifacts into the right format.

[dnicolodi]

Detecting whether --cross-file is present in config_settings is probably reliable (covers everything except for the macOS environment variable way)? If so, could we just read the host machine section of the cross file, and generate the wheel tag from that?

This would require determining which architecture we are building for from the compiler executable paths. I'm sure it can be done, but the user knows for which architecture they are building, they can just tell the wheel packaging tool about it. Also, it would require determining which flavor of python interpreter (cpython/pypy/pyston,... and the relative version) we are building for from the executable path. Also this seems an information that the user is in a better position to tell us.

More problematic is the handling of build dependencies and (optional) build isolation implemented by PEP 517 frontends. You most likely do not want that for cross compiling. Build dependencies for cross compilation need to be correctly handled considering which dependencies are to be run on the host (cython, pythran) and which are libraries for the target (numpy, scipy, ...). PEP 517 frontends cannot do that, and they mostly get in the way.

[rgommers]

the tools that takes a Meson build directory and packs it up in a wheel is not meson-python, but something else, that may or may not share some code and be implemented in the same project as the current meson-python.

This is a little confusing to me. I'm not sure what you have in mind here exactly. Whatever other project it is, I think that is an implementation detail under meson-python. From my perspective only have two things: Meson as the build system, and meson-python as the layer between pip & co to build sdists and wheels. And the goal of meson-python is to make use of Meson by Python projects as seamless as possible.

What is and isn't in PEP 517 isn't that relevant, there's --config-settings as an explicit escape hatch for anything else that's not directly supported by a standard. And that includes cross-compiling. In fact, proper support for cross-compiling is one of the major benefits of moving from distutils & co to Meson. We certainly get a lot of bug reports and questions about it for NumPy and SciPy, and in the past (pre Meson) I've always answered "don't know, distutils doesn't support that, good luck and if you learn something please share". I now want proper support for it.

AFAICT, auditwheel has the same core assumption, thus I don't think it can fix wheels built for an architecture that is not the one where it is being run.

auditwheel was brought up by both of you, but it really isn't relevant. Its job is specifically to deal with distributing wheels on PyPI. There are many packaging systems other than PyPI, so we cannot rely on auditwheel for anything.

that the correct cross-compilation interface is the one provided by Meson, we need a tools that allows access to that interface

It's not really an interface in the sense that meson-python can use it - but I don't think there's a need for that. We only need a few fragments of information. Basically metadata for platform, OS, and interpreter flavor and ABI - I'm not sure that there's much beyond that. So we should just figure that out from the info config_settings incoming data.

This would require determining which architecture we are building for from the compiler executable paths.

Not really - there's a section like this in the cross file:

[host_machine]
system = 'windows'
cpu_family = 'x86_64'
cpu = 'x86_64'
endian = 'little'

it would require determining which flavor of python interpreter (cpython/pypy/pyston,... and the relative version) we are building for from the executable path. Also this seems an information that the user is in a better position to tell us.

That's a good point. I think the requirement here is "build interpreter kind == host interpreter kind" for now (in practice, the main demand is CPython). Possibly there's a need to add the host interpreter to the cross file - let's cross that bridge whne we get to it.

More problematic is the handling of build dependencies and (optional) build isolation implemented by PEP 517 frontends.

No one is going to use build isolation with cross compilation I think. It'll be something like:

python -m build --wheel --no-isolation --skip-dependency-check -C--cross-file=cross_aarch64_linux.ini

There's nothing for us to do here to deal with build isolation AFAIK.

[dnicolodi]

I think the requirement here is "build interpreter == host interpreter" for now

The problem raised in this issue is about the host and the build interpreter being different (if they are the same, of course importlib.machinery.EXTENSION_SUFFIXES needs to be valid for the extension modules being packaged). If the host interpreter is the same as the build interpreter, meson-python already works just fine, AFAIK.

No one is going to use build isolation with cross compilation I think. It'll be something like:

python -m build --wheel --no-isolation --skip-dependency-check -C--cross-file=cross_aarch64_linux.ini

I don't see what going through build and the PEP 517 interfaces gives you in this case. If you need to pass tool-specific command line arguments (the -C--cross-file, which by the way needs to be -Csetup-args=--cross-file=cross_aarch64_linux.ini) you don't even have the advantage of having a tool-agnostic command line interface. Furthermore, you make optional arguments (--no-isolation and --skip-dependency-check) mandatory. It still looks like an hack and a magic incantation more than a solution.

[rgommers]

I think the requirement here is "build interpreter == host interpreter" for now

The problem raised in this issue is about the host and the build interpreter being different

I meant "the same kind, so both CPython or both PyPy". That's a reasonable default, and I think conda-forge's cross compiling jobs for PyPy do that.

I don't see what going through build and the PEP 517 interfaces gives you in this case.

It's the only way to get the .dist-info metadata and a wheel format output that you need. I keep on seeing this confusion, but --no-build-isolation is not niche, it's extremely important (most non-PyPI packagers need this, and I certainly use it more often than not also for local development) and we should treat it on par with the default.

Furthermore, you make optional arguments (--no-isolation and --skip-dependency-check) mandatory. It still looks like an hack and a magic incantation more than a solution.

They're already mandatory for many use cases. --no-isolation was a choice for a default that optimized for "build me a wheel to distribute on PyPI". Many/most other use cases require no isolation. It is definitely not a hack.

[dnicolodi]

It's the only way to get the .dist-info metadata and a wheel format output that you need.

The PEP 517 backend is responsible for generating the .dist-info

meson-python/mesonpy/__init__.py

Lines 556 to 568 in f9dc18f

	def _wheel_write_metadata(self, whl: mesonpy._wheelfile.WheelFile) -> None:
	# add metadata
	whl.writestr(f'{self.distinfo_dir}/METADATA', self._project.metadata)
	whl.writestr(f'{self.distinfo_dir}/WHEEL', self.wheel)
	if self.entrypoints_txt:
	whl.writestr(f'{self.distinfo_dir}/entry_points.txt', self.entrypoints_txt)
	# add license (see https://github.com/mesonbuild/meson-python/issues/88)
	if self._project.license_file:
	whl.write(
	self._source_dir / self._project.license_file,
	f'{self.distinfo_dir}/{os.path.basename(self._project.license_file)}',
	)

It has nothing to do with using a PEP 517 fronend to invoke the wheel packaging tool.

[rgommers]

@dnicolodi I'm not sure what you mean there. meson-python is that backend, and the only way to use meson-python is via a frontend like pip or build.

to invoke the wheel packaging tool.

It seems like you have a conceptual model here that I do not understand. If I understand you correctly, you have something other than Meson and meson-python in mind, but I don't know what that would be.

[eli-schwartz]

From my perspective only have two things: Meson as the build system, and meson-python as the layer between pip & co to build sdists and wheels.

My perspective is a bit different, but I think that's because I approach the whole issue from a different direction

I view meson-python as two things:

• a producer of dist-info metadata, and python distribution (the keyword for "importable thing with metadata) constructor
• binary archiver for pip-compatible installer bundles

And you mention "pip & co" but I think it's a bit simpler, and reduces down to "pip". Or maybe "PyPI".

As I alluded to above, for local builds a wheel is a waste of time, and so are platform tags. What you actually want is a site-packages, and wheels are just a way for build backends to tell installers what files to install. System package managers like conda, dpkg, rpm, pacman, portage, etc. don't care about wheels, because they have better formats that provide crucial cross-domain dependency management among other things. They also, consequently, have better ways to define platform tags than, well, using anything as ill-defined and non-granular as platform tags.

And what even looks at platform tags anyway? Just pip, basically... and, importantly, only in the context of downloading from PyPI.

...

From the perspective of another package manager trying to repackage software produced by the pip package manager, wheels look like those makefiles where "make" creates a tar.gz file that you have to untar, and don't provide a "make install" rule.

auditwheel was brought up by both of you, but it really isn't relevant. Its job is specifically to deal with distributing wheels on PyPI. There are many packaging systems other than PyPI, so we cannot rely on auditwheel for anything.

But this does in fact tie into my belief that platform tags are also specifically to deal with distributing wheels on PyPI.

The rest of the time it is a vestigial organ. While it can't hurt to get it correct where possible, this shouldn't come at the sacrifice of important functionality like generating a successful build+install. When in doubt, apply either a validating placeholder or a genetic tag that is technically correct but provides no information, like "linux" (is that a valid tag? Do you need the CPU architecture?)

The result doesn't matter, if you're locally building the wheel, locally installing it, and locally creating a conda package out of it.

I don't see what going through build and the PEP 517 interfaces gives you in this case. [...] you don't even have the advantage of having a tool-agnostic command line interface.

Because setuptools install had the implementation behavior of a) executing easy_install instead of pip, b) producing egg-info instead of dist-info, and made the unusual development decision to claim that they can't change this because projects might be depending on egg-info specifically, therefore "we will make you stop using egg-info by using bdist_wheel, dropping install, and breaking your project anyway". They then went all-in and declared that they were removing support for interacting with setuptools via a command line.

Because of the privileged position setuptools held in the ecosystem, this has become the new model for build backends, namely, that they shouldn't provide a command line. And ex post facto, this has been reinvented, rather than being due to peculiarities of egg-info, to instead be due to a host of imagined reasons for why command lines are bad, even as an alternative. ;)

The result is that the advantage you get from going via build and a series of command-line arguments is "it's a program that can generate a library API call to a build backend library".

Flit is fighting this trend, as flit_core.wheel provides a command line. However I think the main motivation there is to make it easily bootstrappable (you don't need to build build without having build, before you can build flit_core), not about how noisy the command line is.

It's a weird quirk but ultimately not a huge deal now that the core infrastructure has to some degree settled on flit.

Note that any program which uses PEP 517 to generate a library API call to build a wheel, is a PEP 517 frontend. But not all PEP 517 frontends support build isolation, or default to it. For example, Gentoo Linux has written an internal frontend called gpep517, which is a "Gentoo pep517", that relies on the knowledge that Gentoo never ever wants build isolation.

Yes, we're now seeing a proliferation of incompatible frontend command lines as a retaliatory response to the unification of backend APIs. And no, frontends aren't trivial to write either.

[rgommers]

As I alluded to above, for local builds a wheel is a waste of time, and so are platform tags

Yes and no. All you do with the final zipfile is unpack it and throw it away, so from that perspective it doesn't do much. However, you do need the metadata installed. So a meson install into site-packages gets you a working package, but you're missing the metadata needed to uninstall again. A wheel in this respect is like a filter that ensures everything one needs is present, so uninstalling/upgrading with pip afterwards works, as do things like using importlib.resources.

System package managers like conda, dpkg, rpm, pacman, portage, etc. don't care about wheels

Not as a distribution format, but they do in practice. These tools very often run pip install . --no-build-isolation to build the package. And then as a final step repackage it into a .conda, .rpm or whatever their native format is.

They then went all-in and declared that they were removing support for interacting with setuptools via a command line.

I completely agree that that's a mistake, and I appreciate Meson's nice CLI. But I think that's a mostly unrelated point here. If meson had a command that'd yield the exact same result as pip install . --no-build-isolation then I'd say we could use that and there'd be no need to go through a wheel. But there's no such command (yet, at least).

[eli-schwartz]

Yes and no. All you do with the final zipfile is unpack it and throw it away, so from that perspective it doesn't do much. However, you do need the metadata installed. So a meson install into site-packages gets you a working package, but you're missing the metadata needed to uninstall again. A wheel in this respect is like a filter that ensures everything one needs is present, so uninstalling/upgrading with pip afterwards works, as do things like using importlib.resources.

Right, like I said, meson-python does two things, and one of them is producing that metadata, and the other one is producing that wheel.

I completely agree that that's a mistake, and I appreciate Meson's nice CLI. But I think that's a mostly unrelated point here.

Right, this is very much a side topic in response to @dnicolodi's question about "I don't see what going through build and the PEP 517 interfaces gives you in this case" and magic incantations.

[zboszor]

@zboszor Can you please elaborate on what that is the intent of that patch and why you need it?

If you need to overwrite arguments passed to meson setup, you can just add them to the setup-args: user arguments are passed after the default ones and thus take precedence. One thing that the patch does is to remove the --pefix argument to meson setup, thus when meson-python executes meson install the files are installed into the system python path, to be then copied into the wheel. I'm not sure this is desirable.

This patch is strictly Yocto-specific. The whole meson cross-compiler environment is provided by a meson.bbclass with the native and cross files already created.

The variables passed to meson change from package to package, i.e. the build prefix is package-specific.

It makes little sense to patch pyproject.toml in every package that would use build-backend = mesonpy, because it would mean adapting the patch whenever something changes in this file and the patch may have rejects or fuzz, both are errors in Yocto.

Therefore, passing setup-args on the command line is easier and easier to maintain.

Another problematic thing that the path does is this:

-        r = subprocess.run(list(args), env=self._env, cwd=self._build_dir)
+        r = subprocess.run(' '.join(list(args)).split(), env=self._env, cwd=self._build_dir)

which is pointless at best (when args does not contain any string containing white space).

But it does contain spaces, because there are multiple options.
See the the python_mesonpy.bbclass. meson.bbclass also passes these options to meson in the same order.

PEP517_BUILD_OPTS = '--config-setting=setup-args="${MESONOPTS} ${MESON_SOURCEPATH} ${B} ${MESON_CROSS_FILE} ${EXTRA_OEMESON}"'

Without splitting args here, the whole argument is interpreted as a single string and passed to meson as a single command line argument. meson then tries to interpret the complete string as either source or build directly, which fails.

[zboszor]

I was also puzzled with that patch - I'd think none of that should be necessary, and agree with @dnicolodi's suggestions.

Here is the path in full in formatted form, for ease of reading for others:

diff --git a/meta-python/recipes-devtools/python/python3-meson-python/remove-hardcoded-setup-args.patch b/meta-python/recipes-devtools/python/python3-meson-python/remove-hardcoded-setup-args.patch
new file mode 100644
index 000000000..3edda85dc
--- /dev/null
+++ b/meta-python/recipes-devtools/python/python3-meson-python/remove-hardcoded-setup-args.patch
@@ -0,0 +1,123 @@
--- meson_python-0.13.0.pre0/mesonpy/__init__.py.old	1970-01-01 01:00:00.000000000 +0100
+++ meson_python-0.13.0.pre0/mesonpy/__init__.py	2023-03-13 21:26:52.263117416 +0100
@@ -669,8 +669,6 @@
         self._build_dir = pathlib.Path(build_dir).absolute() if build_dir else (self._working_dir / 'build')
         self._editable_verbose = editable_verbose
         self._install_dir = self._working_dir / 'install'
-        self._meson_native_file = self._source_dir / '.mesonpy-native-file.ini'
-        self._meson_cross_file = self._source_dir / '.mesonpy-cross-file.ini'

These files are not needed, Yocto creates its own native and cross files.

         self._meson_args: MesonArgs = collections.defaultdict(list)
         self._env = os.environ.copy()
 
@@ -679,32 +677,6 @@
         if ninja_path is not None:
             self._env.setdefault('NINJA', str(ninja_path))
 
-        # setuptools-like ARCHFLAGS environment variable support
-        if sysconfig.get_platform().startswith('macosx-'):
-            archflags = self._env.get('ARCHFLAGS')
-            if archflags is not None:
-                arch, *other = filter(None, (x.strip() for x in archflags.split('-arch')))
-                if other:
-                    raise ConfigError(f'Multi-architecture builds are not supported but $ARCHFLAGS={archflags!r}')
-                macver, _, nativearch = platform.mac_ver()
-                if arch != nativearch:
-                    x = self._env.setdefault('_PYTHON_HOST_PLATFORM', f'macosx-{macver}-{arch}')
-                    if not x.endswith(arch):
-                        raise ConfigError(f'$ARCHFLAGS={archflags!r} and $_PYTHON_HOST_PLATFORM={x!r} do not agree')
-                    family = 'aarch64' if arch == 'arm64' else arch
-                    cross_file_data = textwrap.dedent(f'''
-                        [binaries]
-                        c = ['cc', '-arch', {arch!r}]
-                        cpp = ['c++', '-arch', {arch!r}]
-                        [host_machine]
-                        system = 'Darwin'
-                        cpu = {arch!r}
-                        cpu_family = {family!r}
-                        endian = 'little'
-                    ''')
-                    self._meson_cross_file.write_text(cross_file_data)
-                    self._meson_args['setup'].extend(('--cross-file', os.fspath(self._meson_cross_file)))

This chunk is indeed not needed, Yocto builds under Linux.

         # load config -- PEP 621 support is optional
         self._config = tomllib.loads(self._source_dir.joinpath('pyproject.toml').read_text())
         self._pep621 = 'project' in self._config
@@ -749,19 +721,6 @@
             [binaries]
             python = '{sys.executable}'
         ''')
-        native_file_mismatch = (
-            not self._meson_native_file.exists()
-            or self._meson_native_file.read_text() != native_file_data
-        )
-        if native_file_mismatch:
-            try:
-                self._meson_native_file.write_text(native_file_data)
-            except OSError:
-                # if there are permission errors or something else in the source
-                # directory, put the native file in the working directory instead
-                # (this won't survive multiple calls -- Meson will have to be reconfigured)
-                self._meson_native_file = self._working_dir / '.mesonpy-native-file.ini'
-                self._meson_native_file.write_text(native_file_data)

This chunk is needed. The file mismatch handling doesn't work particularly well because the native file is fed externally.
The internally generated cross and native files lines were removed for the same reason.

        # Don't reconfigure if build directory doesn't have meson-private/coredata.data
        # (means something went wrong)
@@ -784,7 +743,7 @@
    def _proc(self, *args: str) -> None:
        """Invoke a subprocess."""
        print('{cyan}{bold}+ {}{reset}'.format(' '.join(args), **_STYLES))
-        r = subprocess.run(list(args), env=self._env, cwd=self._build_dir)
+        r = subprocess.run(' '.join(list(args)).split(), env=self._env, cwd=self._build_dir)

Already explained, setup-args consists of multiple space separated options, which are passed as a single string.

         if r.returncode != 0:
             raise SystemExit(r.returncode)
 
@@ -800,22 +759,6 @@
         """
         sys_paths = mesonpy._introspection.SYSCONFIG_PATHS
         setup_args = [
-            f'--prefix={sys.base_prefix}',

--prefix is indeed passed in by Yocto's settings.

-            os.fspath(self._source_dir),
-            os.fspath(self._build_dir),

Yocto also passes the source and build directories, with the build directory being outside of the source.
What happens if both the source and build directories are specified twice on the meson command line?

-            f'--native-file={os.fspath(self._meson_native_file)}',
-            # TODO: Allow configuring these arguments
-            '-Ddebug=false',
-            '-Doptimization=2',
-
-            # XXX: This should not be needed, but Meson is using the wrong paths
-            #      in some scenarios, like on macOS.
-            #      https://github.com/mesonbuild/meson-python/pull/87#discussion_r1047041306
-            '--python.purelibdir',
-            sys_paths['purelib'],
-            '--python.platlibdir',
-            sys_paths['platlib'],

purelib and platlib are not passed by Yocto and the build can fail because the native python's settings are not what the target build expects.

             # user args
             *self._meson_args['setup'],
         ]
@@ -905,8 +848,7 @@
         editable_verbose: bool = False,
     ) -> Iterator[Project]:
         """Creates a project instance pointing to a temporary working directory."""
-        with tempfile.TemporaryDirectory(prefix='.mesonpy-', dir=os.fspath(source_dir)) as tmpdir:
-            yield cls(source_dir, tmpdir, build_dir, meson_args, editable_verbose)
+        yield cls(source_dir, build_dir, build_dir, meson_args, editable_verbose)

The build directory is created in advance and is different from what meson-python expects here.
This is why it's fed using an environment variable.

    @functools.lru_cache()
    def _info(self, name: str) -> Dict[str, Any]:
@@ -1105,15 +1047,7 @@
        for key, value in config_settings.items()
    }

-    builddir_value = config_settings.get('builddir', {})
-    if len(builddir_value) > 0:
-        if len(builddir_value) != 1:
-            raise ConfigError('Only one value for configuration entry "builddir" can be specified')
-        builddir = builddir_value[0]
-        if not isinstance(builddir, str):
-            raise ConfigError(f'Configuration entry "builddir" should be a string not {type(builddir)}')
-    else:
-        builddir = None
+    builddir = os.environ.get('MESONPY_BUILD')

    def _validate_string_collection(key: str) -> None:
        assert isinstance(config_settings, Mapping)

I hope I explained everything.

I can agree that none of this is needed for a straight build on the host which is also the target system, but Yocto's cross-compiler system needs them.

[eli-schwartz]

• pretty sure you're supposed to pass -Csetup-args=--foo -Csetup-args=--bar so that they are parsed as an array, specifically to avoid string parsing as shell code, which inherently doesn't work because your patch is terminally broken :) it assumes that string.split() is a valid way to tokenize a command line into words. That being said, if yocto happens to know that a yocto bbclass will only use the subset of shell words that are also whitespace-splitted words, this is "safe" (and even more yocto-specific than before). I guess this is awkward because the mesonopts abstraction happens at a more fundamental level than the use of meson-python, and isn't a bash array (??) so it's not trivial to loop over it and do for x in "${opts[@]}"; do newopts+=("-Csetup-args=$x"); done
• the recommended way to set the builddir at the moment is -Cbuilddir=... rather than "try -Csetup-args=... and have meson-python iterate over every setup argument, import meson's argument parser which isn't public API, and determine whether it represents an operand rather than one half of an option-argument"

[zboszor]

Thanks, I will try these.

Still, meson-python shouldn't set --python.purelibdir and --python.platlibdir because that does break cross-compiling.

[dnicolodi]

Still, meson-python shouldn't set --python.purelibdir and --python.platlibdir because that does break cross-compiling.

How does passing this arguments break cross compilation? They only tell Meson where to install some files. Then meson-python picks the files up from the specified location and packs them into a wheel. The content of the wheel does not depend in any way on these arguments.

[dnicolodi]

@zboszor You seem to be patching meson-python for the wrong reasons.

These files are not needed, Yocto creates its own native and cross files.

You can have as many native and cross files as you want. Meson merges them. You don't need to remove the arguments passed to Meson by meson-python to pass your own cross and native files.

This chunk is needed. The file mismatch handling doesn't work particularly well because the native file is fed externally.

If you leave the meson-python generated native file in place, you don't need to remove this code. However, this code will be gone in the next meson-python version.

Already explained, setup-args consists of multiple space separated options, which are passed as a single string.

This is an horrible idea, as @eli-schwartz explained. If you need to pass multiple arguments via setup-args you need to pass them as list. pip till the latest version does not allow this. There is a patch already merged that fixes this, though. If you need to use pip, you can most likely apply that patch to the pip version you are using. pypa/build allows to pass a list of options to setup-args using the -C command line argument multiple times:

python -m build -Csetup-args=-Cfoo=bar -Csetup-args=debug=true -Cbuilddir=/tmp/build3452

--prefix is indeed passed in by Yocto's settings.

The argument to the --prefix setup option should not have any effect on the content of the wheel.

Yocto also passes the source and build directories, with the build directory being outside of the source.

You can pass the build directory to use to meson-python, see the example above. The source directory is passed to meson-python by the Python build front-end (pypa/build or pip or whatever) there is no need to pass it separately.

purelib and platlib are not passed by Yocto and the build can fail because the native python's settings are not what the target build expects.

As already explained, these settings should not have any effect on the content of the generated wheel.

I can agree that none of this is needed for a straight build on the host which is also the target system, but Yocto's cross-compiler system needs them.

I think that none of this is required, the patch is probably based on a misunderstanding of how meson-python works.

[zboszor]

Thank you for all the suggestions and constructive criticism.
The new series is here now, the meson-python recipe doesn't need any patches:
python_mesonpy.bbclass
meson-python recipe

[dnicolodi]

Great! It is nice to see that things works as designed also in an environment very different from the one where we usually test meson-python. Thank you for sticking with us and accepting the criticism!

[h-vetinari]

With the release of meson-python 0.13.0, our scipy builds in conda-forge now broke. They were previously relying on the apparently only accidentally-working circumstance that reconfiguration did not error, but in doing so, it allowed us to set up the compilation as needed, and not get tripped up by the meson-python invocation through scipy.

Of course it would be good to avoid the reconfiguration (and we're trying to do that now in the linked PR), but AFAICT, that runs into the issue that meson-python unconditionally adds a --native-file argument that ends up superseding the --cross-file we've provided. This leads to the weird combination that meson sets up build/host/target correctly:

Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: aarch64
Host machine cpu: aarch64
Target machine cpu family: aarch64
Target machine cpu: aarch64

but will then nevertheless start compiling for x86 on aarch/ppc

[10/1628] Compiling C object scipy/_lib/_test_ccallback.cpython-38-x86_64-linux-gnu.so.p/src__test_ccallback.c.o
[11/1628] Linking target scipy/_lib/_test_ccallback.cpython-38-x86_64-linux-gnu.so
[12/1628] Compiling C object scipy/_lib/_fpumode.cpython-38-x86_64-linux-gnu.so.p/_fpumode.c.o
[13/1628] Linking target scipy/_lib/_fpumode.cpython-38-x86_64-linux-gnu.so

It's completely possible that I misdiagnosed what's going on, but if I'm not too far off, it would be nice to have a way to switch off the automatic --native-file addition. From the docs it appears that it's only possible to append to them.

[rgommers]

@h-vetinari I think you diagnosed it correctly. This seems easy to fix by specifying the desired python binary also in a cross file, since Meson will prefer the cross binary over the native binary for a cross build when both are specified:

(scipy-dev) $ cat native-file.txt 
[binaries]
python = '/home/rgommers/mambaforge/envs/scipy-dev/bin/python'

(scipy-dev) $ cat cross-file.txt 
[binaries]
python = '/home/rgommers/mambaforge/envs/cross-env/bin/python'

(scipy-dev) $ meson setup buildc --cross-file=cross-file.txt --native-file=native-file.txt
...
Program python found: YES (/home/rgommers/mambaforge/envs/cross-env/bin/python)

  User defined options
    Cross files : cross-file.txt
    Native files: native-file.txt

Note that cross files "layer", so specifying an extra cross file with only the python binary in it will not interfere with the other cross file that is obtained from the conda-forge infra.

[dnicolodi]

but will then nevertheless start compiling for x86 on aarch/ppc

This is not what I think is happening. Most likely Meson is using the cross compilation toolchain to compile binaries for aarch64 but naming the Python extension modules for the Python interpreter that meson-python specified in the native file.

PEP 517 and related PEP specify that the build needs to happen for the Python interpreter that is executing the Python build backend. meson-python uses the native-file for instructing Meson to compile for the interpreter that is executing meson-python. Meson uses introspects (executing some code with it) the interpreter and determines compilation flags and extension module filenames. meson-python similarly introspects the interpreter and determines the wheel tags. The wheel tags need to agree with the extension modules file names (and of course with the binary objects in these).

Unfortunately, there is no specification for cross-compilation, other than running the build backend with the target Python interpreter (via emulation or something). Any other solution works against the specification and is destined to break from time to time as the build tools evolve.

The solution proposed by @rgommers works (I think, I haven't tested it) for instructing Meson to build for the Python interpreter specified in the cross file. However, meson-python still introspects interpreter that is executing meson-python to generate the wheel tags, thus, if no other trick is applied (there are undocumented environment variables that affect what the introspection return just enough to trick meson-python to do what is expected in cross-compilation, at least on macOS and on Linux), the wheel tags will be off.

If anyone figures out a reliable way to make all this work, I'll be very interested in adding the solution to the docs, and possibly in a test case on our CI infrastructure, so that we can at least be aware of things breaking.

[h-vetinari]

The solution proposed by @rgommers works (I think, I haven't tested it) for instructing Meson to build for the Python interpreter specified in the cross file. However, meson-python still introspects interpreter that is executing meson-python to generate the wheel tags

This might be a silly question (as I don't understand the whole design space here; if so, apologies), but since meson proper has cross-compilation figured out, why not piggyback on that in meson-python?

As in: require that (when meson-python sees a cross-file) that it contain python under [binaries], and then use that to generate the wheel tags (and ignore the native-file bits). That would seem a consistent API design to me.

The absence of standards in this case might be a good thing for once, because that means you're not breaking anything or anyone by building on top of mesons cross-compilation interface. If a python standards ever appears, we could figure out how to transition to that, but that's rather pie in the sky right now, as opposed to the very real need to cross-compile python packages today. :)

[rgommers]

This might be a silly question (as I don't understand the whole design space here; if so, apologies), but since meson proper has cross-compilation figured out, why not piggyback on that in meson-python?

While cross compilation support is robust in general in Meson, unfortunately it isn't yet fully figured out for Python specifically. Before making more changes, I'd really like to see it finalized and documented in Meson itself: mesonbuild/meson#7049 (comment)

[andyfaff]

In andyfaff/scipy#51 I'm trying to cross compile scipy with build:x86_64 and host:aarch64. The idea is to do a two step CI run on cirrus-ci. The first step is to build a wheel on x86_64 then test natively on aarch64 (both Ubuntu).

I can get scipy to complete the build step. I found I had to set _PYTHON_PLATFORM_HOST to get the correct wheel name.
However, all the shared libs have x86_64 in the extension suffix, i.e. _test_deprecation_call.cpython-310-x86_64-linux-gnu.so instead of _test_deprecation_call.cpython-310-aarch64-linux-gnu.so (i.e. mesonbuild/meson#7049). When I run file *.so on the shared libs within the unpackaged wheel they are all reported as having aarch64 architecture.

When I install the wheel on linux_aarch64 the tests don't work because all the .so files from the wheel aren't loadable.

I tried providing the python executable in the [binaries] section of the cross and host files, but the meson configure process doesn't like that, probably because the host Python is aarch64 and won't run on x86_64.

meson.build:23:13: ERROR: <PythonExternalProgram 'python' -> ['/root/miniconda3/envs/host-env/lib/python3.10']> is not a valid python or it is missing distutils

I know this cross-compilation process is still being worked on. If you have any tips for how I can proceed, or you would like to use that PR link to try and advance cross-compilation, I'm willing to work on it.

EDIT: you can see a build log here, which also has a build artefact associated with it.

[rgommers]

I tried providing the python executable in the [binaries] section of the cross and host files, but the meson configure process doesn't like that, probably because the host Python is aarch64 and won't run on x86_64.

Yes, this is the exact problem I was running into and reported on in detail in mesonbuild/meson#7049. This does work when using crossenv (as conda-forge does), but not without it. We're kind of stuck for the moment on the non-crossenv case until that issue gets sorted out.

For more context, it is possible to get a non-crossenv cross build to work, but it requires doing something like this (quote from scipy/scipy#16783):

On Void Linux, we set a lot of environment variables to tell the host Python to use the sysconfig data for the build arch and also add the build root to PYTHONPATH, allowing the host Python to find these modules and grabbing relevant information (field sizes, shlib suffixes, etc.) for the build arch rather than the host.

[virtuald]

... so, did anything change for this?

[rgommers]

... so, did anything change for this?

Sorry not quite. There was an attempt to fix things in Meson (mesonbuild/meson#12190), which wasn't quite in the right direction. What we are really waiting for is for PEP 739 to be finished, that's basically the one thing we need (and I did review that and ensured it includes everything we need for Meson). We could actually implement support for the current, preliminary static file format in Meson and go with that. That's "just work", but it hasn't been done yet.

I'll note that cross-compiling can be made to work, since Conda-forge and various Linux distros use it for packages like NumPy, SciPy and Matplotlib. It just doesn't have the kind of nice UX that we want for this issue and for official/documented support in meson-python.