py 1.8.2 breaks collect-only package printing on windows

[kmura-dlb]

When having a tests directory that contains subfolders with each containing some tests:
e.g.

tests
  folder_1
  folder_2
    subfolder_1
      test_module.py

collect only always only reports Package as tests
e.g.:

<Package tests>
  <Module test_module.py>
    <Function test_someting[parameter]>

1.8.1 on windows and 1.8.2 on Darwin / linux would do this:
e.g.:

<Package tests\folder_2\subfolder_1>
  <Module test_module.py>
    <Function test_someting[parameter]>

[bluetech]

Do this literal example reproduce?

I'm asking because the change in py 1.8.2 has to do with case-insensitivity in paths, and in your example everything is lower case. So just want to make sure before I examine this issue.

[kmura-dlb]

Here's a zip file with folder structure that can reproduce this.
This is something artificial, but my workspace that triggered this issue did not contain any uppercase characters.
py_issue.zip

[bluetech]

Package only appears when there is an __init__.py somewhere but your zip doesn't have any.

[kmura-dlb]

When I moved the files from windows to Mac it omitted the init files.
Here's the version with inits included (I was testing with inits)
py_issue_init.zip

[kmura-dlb]

There's also comparison of 1.8.1 and 1.8.2 output of pytest tests --collect-only

[bluetech]

I tried to reproduce this what I have available but failed. When I get a chance I'll try it on a windows machine, but until then if someone can provide a test case that would fail on pytest CI that would be ideal.

In any case, this bug likely belongs in pytest, not in py, because the py patch fixes a real correctness bug, but maybe something in pytest relied on the broken behavior...

[kmura-dlb]

Just to clarify - this happens only on windows, so if you tried reproducing on different platform it would not happen.

Could you point to where such a test case could go? I am not sure if I will find the time, but I could try adding this test case to pytest CI.

[bluetech]

You can start by cloning the pytest git repository to your computer, apply the following diff:

diff --git a/testing/test_terminal.py b/testing/test_terminal.py
index f1481dce5..e1758fa57 100644
--- a/testing/test_terminal.py
+++ b/testing/test_terminal.py
@@ -424,6 +424,30 @@ class TestCollectonly:
             ]
         )
 
+    def test_collectonly_windows_bug(self, testdir: Testdir) -> None:
+        p = testdir.makepyfile(
+            **{
+                "__init__": "",
+                "tests/__init__": "",
+                "tests/folder_1/__init__": "",
+                "tests/folder_2/__init__": "",
+                "tests/folder_2/subfolder_1/__init__": "",
+                "tests/folder_2/subfolder_1/test_module": """
+                    def test_something(): pass
+                """,
+            }
+        )
+        result = testdir.runpytest("--collect-only")
+        assert result.ret == 0
+        result.stdout.fnmatch_lines(
+            [
+                "<Package subfolder_1>",
+                "  <Module test_module.py>",
+                "    <Function test_something>",
+            ],
+            consecutive=True,
+        )
+
     def test_collectonly_error(self, testdir):
         p = testdir.makepyfile("import Errlkjqweqwe")
         result = testdir.runpytest("--collect-only", p)

and run this test from the pytest repo like this:

tox -e py -- testing/test_terminal.py -k test_collectonly_windows_bug

Then modify it until it demonstrates a difference of behavior between py==1.8.1 and py==1.8.2. You can switch between versions like this:

.tox/py/bin/pip install py==1.8.1

[nicoddemus]

Actually I get the same behavior on 1.8.1, which leads me to believe the problem is not clear (at least to me).

This is what I get in 5.4.3:

λ pytest .tmp\py_issue\tests --collect-only
======================== test session starts ========================
platform win32 -- Python 3.7.6, pytest-5.4.3, py-1.8.0, pluggy-0.13.1
rootdir: d:\projects\pytest\.tmp\py_issue, inifile: pytest.ini
plugins: hypothesis-5.14.0, forked-1.1.3, xdist-1.33.0
collected 1 item
<Package D:\projects\pytest\.tmp\py_issue\tests\folder_2\subfolder_1>
  <Module test_module.py>
    <Function test_something[parameter]>

======================= no tests ran in 0.04s =======================

And here in 6.0.0/master/py-1.8.0:

λ pytest .tmp\py_issue\tests --collect-only
======================== test session starts ========================
platform win32 -- Python 3.7.6, pytest-5.4.3, py-1.8.0, pluggy-0.13.1
rootdir: d:\projects\pytest\.tmp\py_issue, configfile: pytest.ini
plugins: hypothesis-5.14.0, forked-1.1.3, xdist-1.33.0
collected 1 item

<Package subfolder_1>
  <Module test_module.py>
    <Function test_something[parameter]>

======================= no tests ran in 0.03s =======================

And I get the exact same output in 6.0.0/master and py-1.8.1, py-1.8.2, and py-1.9.0.

(My comment in #7477 (comment) was incorrect).

Notice that the same packages are still being collected, only their name has changed, from "D:\projects\pytest\.tmp\py_issue\tests\folder_2\subfolder_1" to "subfolder_1". This was done intentionally in #7046, for consistency with how all nodes are named.

So I'm not sure, it seems everything is working as intended.

@KrzysztofMurawski can you elaborate on what problem you are having?

[mdmintz]

@nicoddemus You might also want to ask the Microsoft VS Code people who ran into the issue here: microsoft/vscode-python#12372

[nicoddemus]

Managed to understand the problem, but not sure how to proceed: microsoft/vscode-python#12372 (comment)

[bluetech]

So whatever the problem was it will be fixed in pytest-6.0.0.

I did not think of this possibility, that explains why no matter what I tried I couldn't reproduce it.

It will be interesting to bisect this, not sure I will get to it myself though (maybe).

[nicoddemus]

Closing for now as this has been fixed in master already. Let's see if this becomes a more serious problems to warrant a 5.4.4 release.