How to run & debug OpenVINO's pytest?

[usstq]

We sometime meet pytest fail in OpenVINO CI, but we don't know how to reproduce and fix it locally.

[usstq]

There is a doc for that:

https://github.com/openvinotoolkit/openvino/blob/master/src/bindings/python/BUILDING.md

What I did

make sure cmake has correct settings:

$ cd build
$ cmake -LA -N .. | grep ENABLE_PYTHON
$ cmake -LA -N .. | grep ENABLE_OV_ONNX_FRONTEND
$ cmake -LA -N .. | grep CMAKE_INSTALL_PREFIX
# add ENABLE_PYTHON=ON ENABLE_OV_ONNX_FRONTEND=ON CMAKE_INSTALL_PREFIX=./install if needed

create virtual env, install dependency, run pytest

# pip3 install virtualenv
$ python3 -m virtualenv ov
$ source ./ov/bin/active

# install openvino pybinding in virtual env (after make install -j8)
(ov) $ source ./install/setupvars.sh
(ov) $ cd "${OPENVINO_BASEDIR}/src/bindings/python"    # this must be current folder, pytest will try to find tests in cwd
(ov) $ pip3 install -r requirements.txt
(ov) $ pip3 install -r requirements_test.txt
(ov) $ pytest tests -v -k 'not _cuda' --ignore=tests/test_onnx/test_zoo_models.py --ignore=tests/test_inference_engine

using -k "pattern" to run only tests with name matching the pattern.

legacy tox based tests

https://stackoverflow.com/questions/54873637/how-to-run-tests-with-tox-ini

using tox directly will be very heavy since it would setup a whole new virtual env (including recompiling the whole project for generating python whl). so we re-use the existing virtual env, in tox.ini we can see lines started with pytest, which is our command line to run in our virtual env (please ignore flake8 which's only for coding style checks).

pytest --backend="CPU" tests -v -k 'not _cuda' --ignore=tests/test_onnx/test_zoo_models.py --ignore=tests/test_inference_engine
pytest --backend="CPU" tests_compatibility -v -k 'not _cuda' --ignore=tests_compatibility/test_onnx/test_zoo_models.py
pytest --backend="CPU" tests/test_onnx/test_zoo_models.py -v -n 4 --forked -k 'not _cuda' --model_zoo_xfail

[usstq]

Debug pytest

If we only want to debug the C++ part invoked by a python script, than the simplest way is using gdb:

gdb --args python3 `which pytest` tests -v -k 'not _cuda' --ignore=tests/test_onnx/test_zoo_models.py --ignore=tests/test_inference_engine

or

gdb python3
$ run /path/to/pytest tests -v -k 'not _cuda' --ignore=tests/test_onnx/test_zoo_models.py --ignore=tests/test_inference_engine