wrong exit status for Windows CI

[Borda]

Describe the bug
We are running Github Action on multiple OS and for all, we follow a common pipeline to test python code and collect coverage
https://github.com/PyTorchLightning/pytorch-lightning/blob/b6ab7ca1215d32924beb76f6b214171e4e9c5869/.github/workflows/ci-testing.yml#L125
Now we got to case that some tests are failing on Windows but for some reason, it was not coughed by CI, see here block "Test"
https://github.com/PyTorchLightning/pytorch-lightning/pull/2356/checks?check_run_id=805809911
When I remove coverage and keep only pytest it marks the CI test correctly as failing, see Lightning-AI/pytorch-lightning#2358

To Reproduce
How can we reproduce the problem? Please be specific. Don't just link to a failing CI job. Answer the questions below:

1. What version of Python are you using?
   • Python 3.7
2. What version of coverage.py are you using? The output of coverage debug sys is helpful.
   • coverage 5.1
3. What versions of what packages do you have installed? The output of pip freeze is helpful.
   • pytest 5.4.3
   • pytest-cov 2.10.0
   • pytest-flake8 1.0.6
4. What code are you running? Give us a specific commit of a specific repo that we can check out.
5. What commands did you run?
   • coverage run --source pytorch_lightning -m py.test pytorch_lightning tests -v

Expected behavior

marked the CI check as falling if pytest fails

[nedbat]

I don't know what change you are asking for. Coverage exits with the status of the module you are running. Since you are running "coverage run -m pytest", your command should exit with the same status as pytest.

[Borda]

that is not true according to the GitHub actions for Windows as you can see with coverage` pass successfully and without it shows correctly that fails as several tests failed... So I am suspicions if the status passing from module works for Windows and/or there is another limitation from Github actions?

[nedbat]

I haven't used GitHub Actions at all, but this page (https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#exit-codes-and-error-action-preference) makes it sound like on Windows, perhaps only the final command status is used, and earlier failures in a sequence are ignored. Is that possible?

For example, if you remove the coverage report line, then the pytest command will be last, and we can see if the Action fails correctly.

[Borda]

maybe but that would sound that it is not very consistent behaviour across OS... :/

[nedbat]

I agree with you, but shells are different... Can you try removing the report line?

[Borda]

ok, let's see in this draft PR Lightning-AI/pytorch-lightning#2441

EDIT: it seems to help, thx