Support external tests, with __init__, WITHOUT forcing src/ layout

[njsmith]

In the pytest docs, it recommends against the following layout:

setup.py
mypkg/
    ...
tests/
    __init__.py
    foo/
        __init__.py
        test_view.py
    bar/
        __init__.py
        test_view.py

This is a very convenient layout, so it's unfortunate that it's not supported. The main problem is a technical one. As the docs say:

But now this introduces a subtle problem: in order to load the test modules from the tests directory, pytest prepends the root of the repository to sys.path, which adds the side-effect that now mypkg is also importable.

And this is bad because now you can't test against an installed version of mypkg.

My proposal is: we should fix this technical issue. Pytest should be able to import the tests directory, without the side-effect of making mypkg automatically importable as well.

On Python 3, this is actually trivial to implement. All you have to do is:

from importlib.machinery import PathFinder

@attr.s
class FindTopLevelPackageInDir:
    package = attr.ib()
    dir = attr.ib()

    def find_spec(self, fullname, path=None, target=None):
        if fullname == self.package:
            assert path is target is None
            return PathFinder.find_spec(fullname, path=self.dir)

sys.meta_path.insert(0, FindTopLevelPackageInDir("tests", "/some/path/"))

Now import tests, or import tests.foo, will find our tests in "/some/path", ignoring sys.path. But, import mypkg will continue to search sys.path as normal.

It should be possible on python 2 as well, but I haven't yet stared at PEP 302 for long enough to figure out the details.

Anyway, the question is: given that this is technically possible (and pretty easy at that), is it viable to fix this in pytest, or are there other issues that would block that?

[RonnyPfannschmidt]

based on timelines i expect to happen we can address this after pytest5.x which will be the last to support python2

this would also be a great chance to move our python import mechanism away from py.path.local and onto pathlib

we still have to take a look at whether this can nicely integrate assertion rewriting

[njsmith]

Like I said, I'm pretty sure it is possible to do this on python 2 as well. (In particular, sys.meta_path does exist!) It's just different and more poorly documented, so I figured I'd see how the discussion went here before digging into the details.

...oh, doing a bit more grepping, I see that you're already messing with sys.meta_path in order to support assertion rewriting! It seems likely that you could support both at the same time? You might even be able to keep the two hacks separate – a nice thing about how the import system works is that if you intercept the top-level package, then all future sub-packages get loaded directly from the top-level package's __path__, rather than looking at sys.path, and your assertion rewrite code already knows how to handle that. So if you don't care about rewriting assertions in tests/__init__.py, then you could use one hook to find tests/ and then fall back on your regular hooks for everything after that.

[RonnyPfannschmidt]

from my pov, im more than happy/willing to make this a python3 only feature

but we do have to take into account nested varied import root situations
as far as i can tel lthe python3 implementation looks really simple, but the fallout it can create on top of more interesting directory layouts is absolutely unclear, i'll take a bit before i can generate a map of the structural pitfalls that i am under the impression may exist here

[njsmith]

Yeah, I actually have no idea how pytest's logic for test discovery and path management really works, I always just flail around until something useful happens. So I guess that's my biggest question here, whether this could be slotted into that logic without breaking everything.

[bluetech]

This was implemented in #7246.