Distinct exit code for collection errors

[blueyed]

Stopping on collection errors was introduced in #1628 (via #1421), but it uses session.Interrupted, which results in exit code 2 (and this means "interrupted by the user", handled as KeyboardInterrupt even):

Exit code 0: All tests were collected and passed successfully
Exit code 1: Tests were collected and run but some of the tests failed
Exit code 2: Test execution was interrupted by the user
Exit code 3: Internal error happened while executing tests
Exit code 4: pytest command line usage error
Exit code 5: No tests were collected

Example traceback:

[7]   …/Vcs/cosmic-ray/plugins/test-runners/pytest/cosmic_ray_pytest_runner/runner.py(49)_run()
-> exit_code = pytest.main(args, plugins=[collector])
[8]   …/Vcs/pytest/_pytest/config.py(58)main()
-> return config.hook.pytest_cmdline_main(config=config)
[9]   …/Vcs/pytest/_pytest/vendored_packages/pluggy.py(745)__call__()
-> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
[10]   …/Vcs/pytest/_pytest/vendored_packages/pluggy.py(339)_hookexec()
-> return self._inner_hookexec(hook, methods, kwargs)
[11]   …/Vcs/pytest/_pytest/vendored_packages/pluggy.py(334)<lambda>()
-> _MultiCall(methods, kwargs, hook.spec_opts).execute()
[12]   …/Vcs/pytest/_pytest/vendored_packages/pluggy.py(614)execute()
-> res = hook_impl.function(*args)
[13]   …/Vcs/pytest/_pytest/main.py(141)pytest_cmdline_main()
-> return wrap_session(config, _main)
[14]   …/Vcs/pytest/_pytest/main.py(112)wrap_session()
-> session.exitstatus = doit(config, session) or 0
[15]   …/Vcs/pytest/_pytest/main.py(148)_main()
-> config.hook.pytest_runtestloop(session=session)
[16]   …/Vcs/pytest/_pytest/vendored_packages/pluggy.py(745)__call__()
-> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
[17]   …/Vcs/pytest/_pytest/vendored_packages/pluggy.py(339)_hookexec()
-> return self._inner_hookexec(hook, methods, kwargs)
[18]   …/Vcs/pytest/_pytest/vendored_packages/pluggy.py(334)<lambda>()
-> _MultiCall(methods, kwargs, hook.spec_opts).execute()
[19]   …/Vcs/pytest/_pytest/vendored_packages/pluggy.py(614)execute()
-> res = hook_impl.function(*args)
[20] > …/Vcs/pytest/_pytest/main.py(164)pytest_runtestloop()
-> "%d errors during collection" % session.testsfailed)
(Pdb++) l
159  	
160  	def pytest_runtestloop(session):
161  	    if (session.testsfailed and
162  	            not session.config.option.continue_on_collection_errors):
163  	        raise session.Interrupted(
164  ->	            "%d errors during collection" % session.testsfailed)
165  	
166  	    if session.config.option.collectonly:
167  	        return True
168  	
169  	    for i, item in enumerate(session.items):

Should there be a distinct error code for this?

I think it makes sense, since "interactively interrupted / KeyboardError" is really something else than "errors due to imports etc".

Could exit code 3 be used for it also ("internal error")?

/cc @omarkohl

[robert-cody]

Also there is pytest.exit() function that can't be distinguished from KeyboardInterrupt exit - have the same exit code 2 which isn't convenient.

[RonnyPfannschmidt]

die to how exit codes are used in real-world deployments, we are in a bit of a bad position,
i believe we need a major release to shake up the exit codes