buildPythonPackage: Add support for running tests in a separate derivation

[adisbladis]

Description of changes

Introduces a new argument to buildPythonPackage called separateChecks.

A passthru.tests.python attribute is added where the tests are run separately from building the package.

By removing check inputs from the main derivation we can drastically reduce build closures & reduce issues with circular dependencies.
For more motivations see the related tracking issue.

This work is a part of #272178.

Implications

With tests no longer running in the same derivation as the build it can be harder to find issues early.

OTOH debug cycles can be cut shorter as you don't need to wait for a package rebuild to find out that a test failed because it lacked networking access.
With tests running separately I expect iteration cycles to be quicker, once the new workflow has been adopted.

passthru.tests is evaluated by OfBorg. Checks for nixpkgs contributions should be fine.
It's possible we may want to add another Hydra attrset so it can find tests, or make Hydra smart enough to explore passthru.tests.

In this PR only one derivation has been converted as a PoC. Locally I have converted many more and also found some issues.
This seems to work out fine for most pure Python packages, but Pandas was problematic. I think that particular issue is because of a bug in pytest that I will take a closer look at.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[adisbladis]

I'm marking this as draft: Not because I don't think it's ready for review but because I think we need to have a discussion around the implications it has to remove python test running from the critical path in nixpkgs.

[FRidh]

passthru.tests is evaluated by OfBorg. Checks for nixpkgs contributions should be fine.
It's possible we may want to add another Hydra attrset so it can find tests, or make Hydra smart enough to explore passthru.tests.

I think it would be useful to introduce a convention such as passthru.tests.package or passthru.tests.source to denote the tests that are part of the source.

[FRidh]

I'm marking this as draft: Not because I don't think it's ready for review but because I think we need to have a discussion around the implications it has to remove python test running from the critical path in nixpkgs.

I think if we introduce such a convention, and then update our release* sets so Hydra pick these up that we're good to go.

[adisbladis]

Rebased & marked ready for review.

[yajo]

I love it!

[ShamrockLee]

Considering the number of rebuilding packages, should this PR target the staging branch instead?

[dotlambda]

One thing I like about packageOverrides is that all tests are run again, which should ensure intercompatibility of the pinned versions. We'd lose this if a package doesn't depend on its test (and if we get rid of packageOverrides).

[adisbladis]

One thing I like about packageOverrides is that all tests are run again, which should ensure intercompatibility of the pinned versions. We'd lose this if a package doesn't depend on its test (and if we get rid of packageOverrides).

This is what I meant by:

I think we need to have a discussion around the implications it has to remove python test running from the critical path in nixpkgs.

I'm not sure what the fix for that is.
One thing I was considering to get some of those benefits is to push the responsibility of aggregating tests to buildPythonApplication/toPythonApplication (together with #272179), though that would only run tests for libraries depended on by applications, not loose libraries.

[yajo]

My suggestion is that the derivation produced by this new implementation adds automatically some passtrhu.test attribute or similar. This way we can simply run those tests if available, but still have no direct dependency over them. If we depend on them, then there's no gain: we'd need to rebuild always too.