Coverage.py fails to omit third-party coverage in a nested venv from the same implicit namespace package #1869

webknjaz · 2024-10-04T14:47:33Z

Describe the bug

I have a case when I get unexpected coverage measurements from a tox-managed virtualenv existing within the source tree. Coveragepy is invoked via pytest-cov but all configuration is in coveragerc so I'm pretty sure this issue belongs here.

This reminds me of #876 and #905. Having stared at 0285af9 and 5c2f614, I concluded that the fixes are still imperfect.

The reproducers in those issues showcased regular non-namespaces packages, which is why it wasn't on anybody's radar, I think. But I'm sure my case is legit too.

In the example below, awx_plugins is the namespace. The dependency is the awx_plugins.interfaces importable and the project itself contains awx_plugins.credentials and awx_plugins.inventory.
The project's files are under src/awx_plugins/{credentials,inventory}/*.py and tests/*.py. The dependecy has src/awx_plugins/interfaces/*.py in the repo, but all are installed side-by-side as .tox/py/lib/python3.13/site-packages/awx_plugins/{credentials,interfaces,inventory}/*.py. .tox/py/lib/python3.13/site-packages/awx_plugins/__init__.py does not exist. .tox/py/lib/python3.13/site-packages/awx_plugins/{credentials,interfaces,inventory}/__init__.py do not exist either.

The project is configured with

[paths]
_site-packages-to-src-mapping =
  src
  */src
  *\src
  */lib/pypy*/site-packages
  */lib/python*/site-packages
  *\Lib\site-packages

which maps all the project-local source files properly (which is visible in the report correctly). The third-party ones remain unmatched and show up with site-packages paths.

There's also

[run]
relative_files = true
source =
  .
source_pkgs =
  awx_plugins

The generic paths and namespaces effectively give me the ability to copy the good coveragerc file over into similar packages of the same namespace and folder layout. This is intentional.

I'm almost sure changing source_pkgs to list awx_plugins.credentials and awx_plugins.inventory would limit what ends up in the report but that's not something I want given that I wish to have a reusable config across an ecosystem of similar projects (i.e. plugins to something else).

I've tested that adding

omit =
  .tox/**

is a functional workaround, following the suggestions in other issues.

But ultimately, I believe that this is a bug fixable within coveragepy and I shouldn't have to keep the omit settings in my configs.

To Reproduce

What version of Python are you using? A range of Python 3.11-3.13, I'm pretty sure the problem is agnostic, though.

What version of coverage.py shows the problem?

👉 click to unfold 👈

$ .tox/py/bin/python -m coverage debug sys
-- sys -------------------------------------------------------
               coverage_version: 7.6.1
                coverage_module: ~/src/github/ansible/awx-plugins/.tox/py/lib/python3.13/site-packages/coverage/__init__.py
                           core: -none-
                        CTracer: available
           plugins.file_tracers: -none-
            plugins.configurers: covdefaults.CovDefaults
      plugins.context_switchers: -none-
              configs_attempted: ~/src/github/ansible/awx-plugins/.coveragerc
                   configs_read: ~/src/github/ansible/awx-plugins/.coveragerc
                    config_file: ~/src/github/ansible/awx-plugins/.coveragerc
                config_contents: b'[html]\nshow_contexts = true\nskip_covered = false\n\n[paths]\n_site-packages-to-src-mapping =\n  src\n  */src\n  *\\src\n  */lib/pypy*/site-packages\n  */lib/python*/site-packages\n  *\\Lib\\site-packages\n\n[report]\n# `fail_under` is set here temporarily until it can be dropped:\nfail_under = 39.27\nskip_covered = true\nskip_empty = true\nshow_missing = true\nexclude_also =\n  ^\\s*@pytest\\.mark\\.xfail\n\n[run]\nbranch = true\ncover_pylib = false\n# https://coverage.rtfd.io/en/latest/contexts.html#dynamic-contexts\n# dynamic_context = test_function  # conflicts with `pytest-cov` if set here\nomit =\n  .tox/**\nparallel = true\nplugins =\n  covdefaults\nrelative_files = true\nsource =\n  .\nsource_pkgs =\n  awx_plugins\n'
                      data_file: -none-
                       python: 3.13.0b2 (main, Jun 23 2024, 13:25:09) [GCC 13.2.1 20240113]
                       platform: Linux-6.6.13-gentoo-dist-x86_64-Intel-R-_Core-TM-_i7-9850H_CPU_@_2.60GHz-with-glibc2.38
                 implementation: CPython
                    gil_enabled: True
                     executable: ~/src/github/ansible/awx-plugins/.tox/py/bin/python
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 31740
                            cwd: ~/src/github/ansible/awx-plugins
                           path: ~/src/github/ansible/awx-plugins
                                 ~/.pyenv/versions/3.13.0b2/lib/python313.zip
                                 ~/.pyenv/versions/3.13.0b2/lib/python3.13
                                 ~/.pyenv/versions/3.13.0b2/lib/python3.13/lib-dynload
                                 ~/src/github/ansible/awx-plugins/.tox/py/lib/python3.13/site-packages
                                 ~/src/github/ansible/awx-plugins/src
                    environment: HOME = ~
                                 PYENV_ROOT = ~/.pyenv
                                 PYENV_SHELL = zsh
                                 PYENV_VIRTUALENV_INIT = 1
                   command_line: ~/src/github/ansible/awx-plugins/.tox/py/lib/python3.13/site-packages/coverage/__main__.py debug sys
         sqlite3_sqlite_version: 3.44.2
             sqlite3_temp_store: 0
        sqlite3_compile_options: ATOMIC_INTRINSICS=1, COMPILER=gcc-13.2.1 20240113, DEFAULT_AUTOVACUUM,
                                 DEFAULT_CACHE_SIZE=-2000, DEFAULT_FILE_FORMAT=4,
                                 DEFAULT_JOURNAL_SIZE_LIMIT=-1, DEFAULT_MMAP_SIZE=0, DEFAULT_PAGE_SIZE=4096,
                                 DEFAULT_PCACHE_INITSZ=20, DEFAULT_RECURSIVE_TRIGGERS,
                                 DEFAULT_SECTOR_SIZE=4096, DEFAULT_SYNCHRONOUS=2,
                                 DEFAULT_WAL_AUTOCHECKPOINT=1000, DEFAULT_WAL_SYNCHRONOUS=2,
                                 DEFAULT_WORKER_THREADS=0, ENABLE_API_ARMOR, ENABLE_BYTECODE_VTAB,
                                 ENABLE_COLUMN_METADATA, ENABLE_DBPAGE_VTAB, ENABLE_DBSTAT_VTAB,
                                 ENABLE_EXPLAIN_COMMENTS, ENABLE_FTS3, ENABLE_FTS3_PARENTHESIS, ENABLE_FTS4,
                                 ENABLE_FTS5, ENABLE_GEOPOLY, ENABLE_HIDDEN_COLUMNS, ENABLE_ICU,
                                 ENABLE_MATH_FUNCTIONS, ENABLE_MEMSYS5, ENABLE_NORMALIZE,
                                 ENABLE_OFFSET_SQL_FUNC, ENABLE_PREUPDATE_HOOK, ENABLE_RBU, ENABLE_RTREE,
                                 ENABLE_SESSION, ENABLE_STMTVTAB, ENABLE_STMT_SCANSTATUS,
                                 ENABLE_UNKNOWN_SQL_FUNCTION, ENABLE_UNLOCK_NOTIFY,
                                 ENABLE_UPDATE_DELETE_LIMIT, HAVE_ISNAN, MALLOC_SOFT_LIMIT=1024,
                                 MAX_ATTACHED=10, MAX_COLUMN=2000, MAX_COMPOUND_SELECT=500,
                                 MAX_DEFAULT_PAGE_SIZE=8192, MAX_EXPR_DEPTH=1000, MAX_FUNCTION_ARG=127,
                                 MAX_LENGTH=1000000000, MAX_LIKE_PATTERN_LENGTH=50000,
                                 MAX_MMAP_SIZE=0x7fff0000, MAX_PAGE_COUNT=1073741823, MAX_PAGE_SIZE=65536,
                                 MAX_SQL_LENGTH=1000000000, MAX_TRIGGER_DEPTH=1000,
                                 MAX_VARIABLE_NUMBER=32766, MAX_VDBE_OP=250000000, MAX_WORKER_THREADS=8,
                                 MUTEX_PTHREADS, SOUNDEX, SYSTEM_MALLOC, TEMP_STORE=1, THREADSAFE=1, USE_URI

What versions of what packages do you have installed?

👉 click to unfold 👈

$ .tox/py/bin/python -m pip freeze        
adal==1.2.7
attrs==24.2.0
awx-plugins-core @ file://~/src/github/ansible/awx-plugins/.tox/.tmp/package/97/awx_plugins_core-0.0.1a5.dev213%2Bg8ea07661f9.d20241004-0.editable-py3-none-any.whl#sha256=3ca7389cb589da068511245e32662a41eefebf9ecfbbdfa7c41025be26f55c12
awx_plugins.interfaces @ git+https://github.com/ansible/awx_plugins.interfaces.git@3a78f59ffe922638f37bf63b4243aac41ed8abd8
azure-core==1.31.0
azure-identity==1.18.0
azure-keyvault==4.2.0
azure-keyvault-certificates==4.8.0
azure-keyvault-keys==4.9.0
azure-keyvault-secrets==4.8.0
boto3==1.35.31
botocore==1.35.31
certifi==2024.8.30
cffi==1.17.1
charset-normalizer==3.3.2
covdefaults==2.3.0
coverage==7.6.1
coverage-enable-subprocess==1.0
cryptography==43.0.1
execnet==2.1.1
hypothesis==6.112.2
idna==3.10
iniconfig==2.0.0
isodate==0.6.1
jmespath==1.0.1
msal==1.31.0
msal-extensions==1.2.0
msrest==0.7.1
msrestazure==0.6.4.post1
oauthlib==3.2.2
packaging==24.1
pluggy==1.5.0
portalocker==2.10.1
pycparser==2.22
PyJWT==2.9.0
pytest==8.3.3
pytest-cov==5.0.0
pytest-mock==3.14.0
pytest-xdist==3.6.1
python-dateutil==2.9.0.post0
python-dsv-sdk==1.0.4
python-tss-sdk==1.2.3
PyYAML==6.0.2
requests==2.32.3
requests-oauthlib==2.0.0
s3transfer==0.10.2
six==1.16.0
sortedcontainers==2.4.0
typing_extensions==4.12.2
urllib3==2.2.3

What code shows the problem? ansible/awx-plugins@8ea0766

What commands should we run to reproduce the problem?

$ python3 -Im pip install tox
$ git clone https://github.com/ansible/awx-plugins.git
$ cd awx-plugins
$ python3 -Im tox
$ python3 -Im webbrowser .tox/py/tmp/htmlcov/index.html

That shows a couple of files under .tox/py/lib/python3.13/site-packages/awx_plugins/interfaces/**. They should not be listed.

Expected behavior

Coveragepy should not collect coverage in third-party modules from the same implicit namespace, installed in the same virtualenv.

Additional context

The concrete example is inspectable at https://app.codecov.io/gh/ansible/awx-plugins/commit/8ea07661f9da09dedaa52b569978c448fb871931/tree?flags%5B0%5D=pytest. The pre-selected pytest flag is what you want to have selected at all times since there's also MyPy coverage that's overlayed if you deselect it.

The text was updated successfully, but these errors were encountered:

This is a workaround for a coveragepy imperfection where it mistakenly treats same-namespace third-party modules as same-project ones: nedbat/coveragepy#1869. Refs: * nedbat/coveragepy#876 * nedbat/coveragepy#905 * nedbat/coveragepy@27d8255 * nedbat/coveragepy@5c2f614

webknjaz · 2024-10-04T15:49:32Z

I've tested that adding
omit =
  .tox/**
is a functional workaround, following the suggestions in other issues.

Nope. It only appeared to be a full workaround, but it's not. It does what I expected in the local HTML/XML reports, but excludes all of the awx_plugins namespace files from the XML report in CI, making Codecov not show anything. I checked that codecov receives XML files with tests/ paths while locally I also have src/ paths. The coveragepy version is the same and other things should be as well, due to how I pin the environment deps. This is very weird...

Ref: nedbat/coveragepy#1869 (comment)

webknjaz · 2024-10-04T16:13:48Z

I ended up changing that to a granular

omit =
  .tox/*/lib/pypy*/site-packages/awx_plugins/interfaces/**
  .tox/*/lib/python*/site-packages/awx_plugins/interfaces/**
  .tox\*\Lib\site-packages\awx_plugins\interfaces\**

for now, but I don't understand why running the same command in the CI and locally would produce different coverage reports. It seems like the environment somehow influences how omit works, which is rather confusing.

webknjaz added the bug Something isn't working label Oct 4, 2024

webknjaz added a commit to ansible/awx_plugins.interfaces that referenced this issue Oct 4, 2024

🧪 Only exclude awx_plugins.interfaces cov

23880da

Ref: nedbat/coveragepy#1869 (comment)

webknjaz added a commit to ansible/awx-plugins that referenced this issue Oct 4, 2024

🧪 Only exclude awx_plugins.interfaces cov

b8b450e

Ref: nedbat/coveragepy#1869 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Coverage.py fails to omit third-party coverage in a nested venv from the same implicit namespace package #1869

Coverage.py fails to omit third-party coverage in a nested venv from the same implicit namespace package #1869

webknjaz commented Oct 4, 2024

webknjaz commented Oct 4, 2024

webknjaz commented Oct 4, 2024

Coverage.py fails to omit *third-party* coverage in a nested venv from the same implicit namespace package #1869

Coverage.py fails to omit *third-party* coverage in a nested venv from the same implicit namespace package #1869

Comments

webknjaz commented Oct 4, 2024

webknjaz commented Oct 4, 2024

webknjaz commented Oct 4, 2024

Coverage.py fails to omit third-party coverage in a nested venv from the same implicit namespace package #1869

Coverage.py fails to omit third-party coverage in a nested venv from the same implicit namespace package #1869