ENH: add support for PEP 639

[dnicolodi]

Closes gh-270

[henryiii]

Looks pretty good from a quick review. I'll keep this in mind when I implement it (very soon) in scikit-build-core.

[henryiii]

Do you have a link to a PR or something for the 1.6 change? curious as to what needed updates in meson, was expecting it to be just meson-python.

[eli-schwartz]

Probably due to depending on mesonbuild/meson#13783

[dnicolodi]

Indeed, as @eli-schwartz says. Meson support is required only to get the license and license files declared in meson.build when they are declared dynamic or when there is no project section in pyproject.toml.

[dnicolodi]

Looks pretty good from a quick review. I'll keep this in mind when I implement it (very soon) in scikit-build-core.

It is a bit uglier than what I would like because I wanted to keep compatibility with older pyproject-metadata. meson-python supports anything back to version 0.7.1.

[QuLogic]

I gave this a shot in matplotlib/matplotlib#28982, and License-Expression is correctly pulled from meson.build, as are License-File from the main project, but what I'd be missing there is License-File from the subprojects.

[dnicolodi]

I'm not very well versed in licensing issues, but I don't think the way PEP 639 supports declaring package licenses works well for this use case. The main problem is that it expects the license of the package's sdist and wheel to be exactly the same. For packages like matplotlib that link statically with some libraries, this is not the case: the license of the source code is the matplotlib license, the license of the wheel is the union of the matplotlib license with the license of all the libraries linked to it.

Concretely, the issue is that in the most common case the subprojects directory does not contain the source code of the subprojects but only references to them (in the matplotlib case in the form of meson wrap files). However, the License-File fields need to point to actual files distributed in the sdist and in the wheel. The sdist in these cases does not contain the license files, thus the License-File fields for the subprojects cannot be added. The metadata information for the sdist and the wheel need to be the same, thus the fields cannot be added to the wheel metadata either.

I don't know how the problem could be solved. The most formally right solution would to allow source and binary distributions to have different licenses. But this is a quite significant change in behavior for the package metadata system.

The pragmatic solution is to add the subprojects' license files to the source distribution and accept that the sdist as a whole might have a more restrictive license than what it could have.

[eli-schwartz]

However, the License-File fields need to point to actual files distributed in the sdist and in the wheel. The sdist in these cases does not contain the license files, thus the License-File fields for the subprojects cannot be added. The metadata information for the sdist and the wheel need to be the same, thus the fields cannot be added to the wheel metadata either.

This is not true. Metadata can be dynamic and vary between the sdist and the wheel. It's already dynamic as a result of being calculated from meson introspection, so you just need to mark that in the sdist as well.

The decision to make sdist metadata static and forbidden to change is... definitely a decision of all time. But the good news is it comes with an escape hatch. This escape hatch is needed since the spec also says that it's illegal per the spec to apply open source patches to an sdist metadata if the fields are marked as static. 🤷

[dnicolodi]

This is not true. Metadata can be dynamic and vary between the sdist and the wheel.

I was convinced that License-Expression and License-File were excluded from being dynamic as Name and Version are. However, scanning through PEP 639, this does not seem to be the case.

It's already dynamic as a result of being calculated from meson introspection, so you just need to mark that in the sdist as well.

It is not dynamic in the sense of PEP 658. It is calculated at sdist build time and not declared as Dynamic in the metadata.

Declaring these fields as dynamic is an interesting solution for the problem at hand, thanks @eli-schwartz for pointing it out. However, how to expose this to the users of meson-python is not really clear to me. I'm not sure that the mere presence of subprojects should be a strong enough indicator that License-Files should be treated as a PEP 658 Dynamic field. This would also raise the question of what to do with License-Expression. Should it also marked Dynamic and have it set to a (root project license) AND (subproject 1 license) AND (subproject 2 license) AND ... equivalent license expression?

[dnicolodi]

Though there's a statement or two about license-files in the PEP that might be incorrect for this sort of usage.

Which ones?

[henryiii]

If a License-File is listed in a Source Distribution or Built Distribution’s Core Metadata:

• That file MUST be included in the distribution archive at the specified path relative to the root license directory.

In my mind, I inserted "Source" here in the first bullet point. I think it's actually correct, though.

[henryiii]

I plan to work on the dynamic-metadata project soon and that would provide a way to do this from a Python hook. Not sure if this would help with integration with Meson, though.

[eli-schwartz]

Declaring these fields as dynamic is an interesting solution for the problem at hand, thanks @eli-schwartz for pointing it out. However, how to expose this to the users of meson-python is not really clear to me. I'm not sure that the mere presence of subprojects should be a strong enough indicator that License-Files should be treated as a PEP 658 Dynamic field.

It's less an "interesting solution for the problem" and more just... the correct way to define it?

It is dynamic because the final value depends on which subprojects end up configured when building a wheel. You can (and should) always mark it as PEP 658 dynamic because meson-python reserves the right to recalculate it later on.

This would also raise the question of what to do with License-Expression. Should it also marked Dynamic and have it set to a (root project license) AND (subproject 1 license) AND (subproject 2 license) AND ... equivalent license expression?

Yes? It's the exact same topic. A subproject can define both the license SPDX expression and the license files. If the wheel build has configured that subproject, it is part of the resulting metadata for the binary distribution.

[dnicolodi]

I think that this reasoning is correct if the subprojects are linked statically. However, if they are linked dynamically, the binary distribution in itself is not bound by the license of the dynamically linked libraries. Is it? What about subprojects that are only used as build tools? Or any other use of subprojects that does not end up in static linking to the distributed binaries?

Similarly, what if subprojects are only fallbacks for dependencies that are not found locally? If the binary distribution links statically to a locally installed library, it is still bound by the terms of the license of said library, but the subproject would not be build.

Yes? It's the exact same topic. A subproject can define both the license SPDX expression and the license files. If the wheel build has configured that subproject, it is part of the resulting metadata for the binary distribution.

The question was more whether concatenating the licenses with a logic AND is always the appropriate thing to do. I think it is, but IANAL.

[eli-schwartz]

I think that this reasoning is correct if the subprojects are linked statically. However, if they are linked dynamically, the binary distribution in itself is not bound by the license of the dynamically linked libraries. Is it? What about subprojects that are only used as build tools? Or any other use of subprojects that does not end up in static linking to the distributed binaries?

If you distribute a dynamically linked library that has been auditwheel'd into your wheel, you most certainly are bound by its license.

dynamically linked libraries have different license terms than statically linked ones, some of the time, such as for example being able to use the LGPL instead of the GPL. Still bound by the license.

What about subprojects that are only used as build tools?

I'm not sure what to say there besides for "difficult to detect... :("

Similarly, what if subprojects are only fallbacks for dependencies that are not found locally? If the binary distribution links statically to a locally installed library, it is still bound by the terms of the license of said library, but the subproject would not be build.

Then meson can't help you report this, but it is still true that those are the terms you must follow. However typical wheel building processes tend to happen on systems where dependencies, if available, are dynamically available -- which means either you distribute it yourself via e.g. auditwheel or require the system to provide one in which case it's technically not your responsibility to list the terms.

The question was more whether concatenating the licenses with a logic AND is always the appropriate thing to do. I think it is, but IANAL.

"AND", because you have to fulfill both terms. The lawyers are the people you ask to find out what "fulfill both sets of requirements" means. :)

[dnicolodi]

This is why I think that the mere presence of subprojects is not a very good indicator for how the license of the resulting wheel needs to be "composed".

[eli-schwartz]

I think you will find that pure build tools without any influence on the resulting compiled software are relatively rare. It is nearly always more correct to compose the license based on in-use subprojects. And you still can't detect the case where a single project has multiple licenses but one of those licenses only applies to build tools.

I see no advantage in excluding this available subproject metadata.

[QuLogic]

IMO automatic determination based on subproject usage is the only reason to care about PEP639's dynamism. It it didn't use that information, then it's basically static, and we'd just not fill in Meson's copy since it'd be unused and just duplicate metadata.

[dnicolodi]

IMO automatic determination based on subproject usage is the only reason to care about PEP639's dynamism

What do you mean? PEP 639 does not define any dynamism. It only puts some order on how license information is recorded in Python package metadata.

What I am trying to say is that basing the definition of the license of the package on the presence of configured subprojects has the potential to produce incorrect results. I believe that the implementing a feature that has the potential of producing incorrect results in some cases and save typing a few hundred characters in the best case is good tradeoff. Therefore, I will not work on it.

However, if someone comes up with a solution that implements dynamic license determination based on the configured subprojects, that can optionally be enabled, and that does not break any of the features already implemented, I will be happy to review and eventually merge it.

[henryiii]

Clear for pyproject-metadata release?

[dnicolodi]

Yes, green light 🙂