Nesting structure of parametrized classes

[altendky]

I have a parametrized class in pytest. I would like the results to be grouped in PyCharm (and maybe for other reports, hopefully this isn't PyCharm specific) by the parametrization values.

• TheClass
  • Value1
    • test_a
    • test_b
  • Value2
    • test_a
    • test_b

Instead I get the methods repeated with the values inside them.

• TheClass
  • test_a
    • Value1
  • test_b
    • Value1
  • test_a
    • Value2
  • test_b
    • Value2

It seemed like perhaps I could do something with pytest_pycollect_makeitem() but I haven't figured it out yet. It also seems like this might be preferred default behavior. Potentially it might even make sense with multiple parametrization to nest them.

• TheClass
  • Param1
    • Value1
      • test_a
      • test_b
    • Value2
      • test_a
      • test_b
  • Param2
    • Value1
      • test_a
      • test_b
        etc...

Thanks for any pointers on how to achieve this or consideration of this as a feature.

---

https://repl.it/@altendky/pytest-breaking-down-assertions-as-results-2

main.py

# i have a parametrized class in pytest.  i would like the results to be grouped in pycharm (hopefully this isn't pycharm specific) by the parametrization values.  <TheClass <Value1 <test_a, test_b>>, <Value2 <test_a, test_b>>> instead of <TheClass <test_a <Value1>>, <test_b <Value1>>, <test_a <Value2>>, <test_b <Value2>>>.

# Desired:
# <TheClass
#     <Value1
#         <test_a, test_b>
#     >,
#     <Value2
#         <test_a, test_b>
#     >
# >

# Present:
# <TheClass
#     <test_a <Value1>>,
#     <test_b <Value1>>,
#     <test_a <Value2>>,
#     <test_b <Value2>>,
# >

import pytest

pytest.main(['--collect-only'])
pytest.main(['-v'])

test_mine.py

import pytest


def parametrize(scenarios):
    idlist = []
    argvalues = []
    for name, kwargs in scenarios.items():
        idlist.append(name)
        items = kwargs.items()
        argnames = [x[0] for x in items]
        argvalues.append(([x[1] for x in items]))
    
    return pytest.mark.parametrize(
        argnames,
        argvalues,
        ids=idlist,
        scope="class",
    )


@parametrize({
    'Value1': {'attribute': 'value_1'},
    'Value2': {'attribute': 'value_2'},
})
class TestClass:
    def test_a(self, attribute):
        pass

    def test_b(self, attribute):
        pass

[pytestbot]

GitMate.io thinks possibly related issues are #1192 (Pytest class scope fixture for a parametrized test called once for each parameter), #199 (Marks don't pick up nested classes), #1931 (Nested conftest.py), #568 (markers stains on all related classes), and #484 (Order of class fixtures).

[RonnyPfannschmidt]

thats a pycharm issue, pytest itself has separate items per parameterset, and no distinction on thise levels

[altendky]

@RonnyPfannschmidt, both have separate items for each parametrized value of each test and my point in regards to pytest is exactly the lack of layering. I understand that PyCharm could do whatever it wants with the data but it seemed like pytest could be applying a bit more structure to the test nodes which would group the parametrization. It seemed a generally useful feature across various reporting mechanisms so I thought it might be helpful to have some piece of the feature implemented in pytest.

(from --collect-only on the example linked above)

<Module 'test_mine.py'>
  <Class 'TestClass'>
    <Instance '()'>
      <Function 'test_a[Value1]'>
      <Function 'test_b[Value1]'>
      <Function 'test_a[Value2]'>
      <Function 'test_b[Value2]'>

Without knowing much of anything about the pytest node structure I vaguely imagine something like this.

<Module 'test_mine.py'>
  <Class 'TestClass'>
    <Instance '()'>
      <Parametrized 'Value1'>
        <Function 'test_a[Value1]'>
        <Function 'test_b[Value1]'>
      <Parametrized 'Value2'>
        <Function 'test_a[Value2]'>
        <Function 'test_b[Value2]'>

[RonnyPfannschmidt]

currently grouping by parameterization is not done as part of the collection tree at all

the item list is reordered in a way where setup/teardown is minimized wrt scopes, as such its not possible to correctly map this into a tree directly

[RonnyPfannschmidt]

(pytest sees a list of tests when doing the running, there is no structure at that level)

[altendky]

@RonnyPfannschmidt, thanks for the explanation. I see how the intent I want needs to be handled independently. Also, I need to review ways to control test ordering. The purpose of my class is to group a sequence of test functions so that I can get a separate result for each step. They presently run in order but it sounds like that's fragile.

Cheers,
-kyle