-
-
Notifications
You must be signed in to change notification settings - Fork 432
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
wrong exit status for Windows CI #1003
Comments
I don't know what change you are asking for. Coverage exits with the status of the module you are running. Since you are running "coverage run -m pytest", your command should exit with the same status as pytest. |
that is not true according to the GitHub actions for Windows as you can see with coverage` pass successfully and without it shows correctly that fails as several tests failed... So I am suspicions if the status passing from module works for Windows and/or there is another limitation from Github actions? |
I haven't used GitHub Actions at all, but this page (https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#exit-codes-and-error-action-preference) makes it sound like on Windows, perhaps only the final command status is used, and earlier failures in a sequence are ignored. Is that possible? For example, if you remove the coverage report line, then the pytest command will be last, and we can see if the Action fails correctly. |
maybe but that would sound that it is not very consistent behaviour across OS... :/ |
I agree with you, but shells are different... Can you try removing the report line? |
ok, let's see in this draft PR Lightning-AI/pytorch-lightning#2441 EDIT: it seems to help, thx |
Describe the bug
We are running Github Action on multiple OS and for all, we follow a common pipeline to test python code and collect coverage
https://github.com/PyTorchLightning/pytorch-lightning/blob/b6ab7ca1215d32924beb76f6b214171e4e9c5869/.github/workflows/ci-testing.yml#L125
Now we got to case that some tests are failing on Windows but for some reason, it was not coughed by CI, see here block "Test"
https://github.com/PyTorchLightning/pytorch-lightning/pull/2356/checks?check_run_id=805809911
When I remove
coverage
and keep onlypytest
it marks the CI test correctly as failing, see Lightning-AI/pytorch-lightning#2358To Reproduce
How can we reproduce the problem? Please be specific. Don't just link to a failing CI job. Answer the questions below:
coverage debug sys
is helpful.pip freeze
is helpful.coverage run --source pytorch_lightning -m py.test pytorch_lightning tests -v
Expected behavior
marked the CI check as falling if pytest fails
The text was updated successfully, but these errors were encountered: