Test Coverage

Setting Up Coverage

Current testing relies on subprocesses to run tests of the Python scripts. This makes measuring test coverage somewhat difficult, as normally a Python test coverage utility such as Coverage will not be able to determine if a subprocesses code is a) Python and b) needs to be measured.

To circumvent this, we can hack a startup script that is run automatically when Python is started. By default, Python will search $PYTHONPATH for any .pth files (Path Configuration Files) to configure the path and import any additional modules. Only lines starting with 'import ' will be executed. We need to import coverage, and we need to run 'coverage.process_startup()'. We can do this with 'import coverage; coverage.process_startup(); print("Running Coverage")'.

This starts Coverage, but we also need to configure it to only check coverage for code we're interested in. Coverage checks the environment variable 'COVERAGE_PROCESS_START' for the location of a configuration file which we can use to specify what code should and should not be measured. In our case, I've simply omitted code in the /usr and /projects directories.

A caveat - Coverage supports performing measurements over multiple runs, but the documentation implies it's unsafe to do so when running multiple processes simultaneously. To prevent this from happening, the parallel option needs to be set to True. This will append a few things like the machine name and PID to the results filename for each process. Note that this will make combining results from different machines easier to give a more complete report of covered code. Coverage also needs to be informed of what multiprocessing library we're using. I'm uncertain we're using any, but saw a reference to multiprocessing, so I've specified it to be safe.

[run]
branch = True
concurrency = multiprocessing
parallel = True
omit =
/usr/*
/projects/*

Generating the Report

To generate the report, we need to combine all of the .coverage.* files by collecting them into a single directory and running coverage combine. Note that if the working directory is changed to create the subprocess, the directory that the .coverage.* file was created in will be changed as well.
We can then generate a report with coverage html.