Python: run tests in separate derivation

[FRidh]

Issue description

Currently we run tests directly after a package has been build. Unfortunately, we need to disable tests regularly to avoid circular dependencies caused by test dependencies.

Running tests in a separate derivation would:

• allow us to test all packages because circular dependencies are no longer a issue;
• reduce build time of the actual packages because tests are ran outside of them;
• add a bit of overhead because there's now two derivations per package;
• allow actual integration testing, because we would build a new env in which we perform the testing.

I propose we then add a tested attribute to Python packages that corresponds to the second derivation. This is typically the attribute to use for CI as well.

[copumpkin]

I've often wanted this for other types of derivations, too, but I'd definitely support this.

[anderspapitto]

One potential snag - there are probably packages where some tests rely on access to temporary build files.
In those cases, you'd either have to disable any such tests, or alternatively the tested attribute could redo the whole build.

[peterhoeg]

Earlier conversation #9281.

[deliciouslytyped]

It's also very annoying during packaging / nix package development when a test fails after an hour long build and you have to go through the whole process again just to work on fixing the tests.
(Ok, you can use nix-shell)

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[doronbehar]

I don't think we should (by default) add such a tested attribute, or put the tests in a passthru.tests derivation. In fact, I think that building the tests in the same derivation has the advantage that it forces us to observe the failure, which in turn can warn us about failures also in dependent packages. The same reasoning also pushes me to advocate in transitioning from adding a passthru.tests.version to versionCheckHook. See also:

• https://nixos.org/manual/nixpkgs/unstable/#tester-testVersion
• versionCheckHook: init #320266
• And the discussion that began here: https://discourse.nixos.org/t/improve-the-jdk-infrastructure-on-nixpkgs/45678/14?u=doronbehar .