Exit pytest if a collection error occours

[fontealpina]

If a collection error occurs, is there a way to terminate immediately pytest execution after collection, avoiding the execution of all the tests (even the ones properly collected)?
Basically I need to start test execution only if there are no import errors.
So for instance, now if there is a collection error I get:

$ pytest test/test_01/test_01.py test/test_02/test_02.py test/test_03/test_03.py -v -s
============================ test session starts =============================
platform linux2 -- Python 2.6.6 -- py-1.4.26 -- pytest-2.7.0.dev1 -- /usr/bin/python
collected 2 items / 1 errors

test/test_01/test_01.py::TestBug::test_n_01 PASSED
test/test_02/test_02.py::TestBug::test_n_02 PASSED

=================================== ERRORS ===================================
________________ ERROR collecting test/test_03/test_03.py _________________
import file mismatch:
a description of the import error
================ 2 passed, 1 error in 0.17 seconds ================

I would get something like this instead:

$ pytest test/test_01/test_01.py test/test_02/test_02.py test/test_03/test_03.py -v -s
============================ test session starts =============================
platform linux2 -- Python 2.6.6 -- py-1.4.26 -- pytest-2.7.0.dev1 -- /usr/bin/python
collected 2 items / 1 errors
=================================== ERRORS =============================
________________ ERROR collecting test/test_03/test_03.py _______________________
import file mismatch:
a description of the import error
================================ 1 error in 0.17 seconds ======================

Is there an options that allows that?
Or maybe this can be done with a plugin?
Thanks

[The-Compiler]

You can easily do this with a local plugin, e.g. by putting this in a conftest.py:

import pytest

def pytest_collectreport(report):
    if report.failed:
        raise pytest.UsageError("Errors during collection, aborting")

Then even if tests are collected, your hook will exit before they are run:

collecting 1 items / 1 errors
================================================ ERRORS =================================================
_____________________________________ ERROR collecting test_two.py ______________________________________
test_two.py:1: in <module>
    import asifjsafpo
E   ImportError: No module named 'asifjsafpo'
======================================== 1 error in 0.01 seconds ========================================
ERROR: Errors during collection, aborting

[fontealpina]

Great, it works as I was expecting!
Thanks

[nicoddemus]

@fontealpina would you consider writing a short blog post with this trick in pytest-tricks? 😉

cc @hackebrot

[fontealpina]

@nicoddemus it is a pleasure for me. I'll do it as soon as possible.

[nicoddemus]

Nice, thanks! 😎

[hpk42]

Actually i think it's worthwhile to consider if we want to consider changing the default to stop if a collection error occurs. Having good defaults is a major goal IMHO. Who ever likes to execute tests when there are collection errors? Whenever i see them i usually want to first see what's going wrong with collection before considering anything else. If there is a real need we can introduce a --continue-on-collection-errors option or so.

[The-Compiler]

Hmm, I actually agree with that - let's reopen this

[nicoddemus]

I agree... if I see collection errors the first thing I do is to immediately hit CTRL+C to stop the run. 😁

[RonnyPfannschmidt]

since the collection behavior is a "breaking" change, we should do it on a major release

[nicoddemus]

IMHO this could go into a 2.10.0 release... it is a small behavior change, and to restore the previous behavior (which I doubt there are people depending on) is just a matter of adding a line to pytest.ini:

[pytest]
addopts = --continue-on-collection-errors

What IMHO should definitely go into a major release are API changes which break entire test suites.

[RonnyPfannschmidt]

we are certain to break behind a firewall ci setup in a minor release then

[nicoddemus]

Sorry, what do you mean by "firewall ci setup"?

[RonnyPfannschmidt]

as in we are certain to "break" testsuites in private setups that previously ran in spite of a few minor collection errors

[nicoddemus]

But those test suites were failing to anyway, no? 😉 After all a collection error does fail a test suite.

[hpk42]

i think the compat aim of pytest minor releases (like 2.10 compared to 2.9) is that all previously passing test suites will still pass. Wouldn't make too big promises on details such as the one we are discussing.

[ghostsquad]

why does pytest exit with code 0 when there is a collection error? I'm using --strict but it doesn't seem to have any effect on the exit code (which is needed for Continuous Build/Integration)

============ 16 passed, 1 pytest-warnings, 1 error in 1.72 seconds =============

Process finished with exit code 0

[ghostsquad]

oh weird, this seems to only occur in PyCharm....

$ python3 -m pytest
...
16 passed, 1 pytest-warnings, 1 error in 0.65 seconds 
$ echo $?
1

[nicoddemus]

@ghostsquad which version are you using? I just did a quick test here and things seem to work as expected:

# contents of foo.py
import kjanskjan
def test_foo():
    pass

# contents of bar.py
def test():
    pass

$ py.test foo.py bar.py -q
.
=================================== ERRORS ====================================
___________________________ ERROR collecting foo.py ___________________________
foo.py:1: in <module>
    import kjanskjan
E   ImportError: No module named 'kjanskjan'
1 passed, 1 error in 0.02 seconds

$ echo %ERRORLEVEL%
1

(Tested in pytest 2.9.0 and 2.8.7)

[ghostsquad]

@nicoddemus ya, it seems fine on the terminal, but within PyCharm CE 5.0.4, it is all "green" and exit code 0. py.tests configuration in PyCharm is default with options: --verbose --strict

I might have to bring this up to Jetbrains.