Apparently inaccurate coverage under pypy3.8-v7.3.6-linux64

[jepler]

Describe the bug
Coverage reports a missed branch when run with pypy3.8-v7.3.6-linux64, while giving 100% coverage in python3.6 through 3.10 and pypy3.7-v7.3.5.

I suspect this may be a pypy bug but I raise it here first because it manifests as a failure in coverage testing, and you may have a better idea how to create a reproducer or what kind of data to pass to the pypy team.

To Reproduce

$ /tmp/pypy3.8-v7.3.6-linux64/bin/pypy -mcoverage debug sys
-- sys -------------------------------------------------------
               coverage_version: 6.0.2
                coverage_module: /tmp/pypy3.8-v7.3.6-linux64/lib/pypy3.8/site-packages/coverage/__init__.py
                         tracer: -none-
                        CTracer: unavailable
           plugins.file_tracers: -none-
            plugins.configurers: -none-
      plugins.context_switchers: -none-
              configs_attempted: .coveragerc
                   configs_read: /home/jepler/src/wwvbpy/.coveragerc
                    config_file: /home/jepler/src/wwvbpy/.coveragerc
                config_contents: b'# SPDX-FileCopyrightText: 2021 Jeff Epler\n#\n# SPDX-License-Identifier: GPL-3.0-only\n[run]\nomit =\n    */site-packages/*\n    test*.py\n'
                      data_file: -none-
                         python: 3.8.12 (279d80ac2079, Oct 17 2021, 05:25:30)[PyPy 7.3.6 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]
                       platform: Linux-5.10.0-8-amd64-x86_64-with-glibc2.2.5
                 implementation: PyPy
                     executable: /tmp/pypy3.8-v7.3.6-linux64/bin/pypy
                   def_encoding: utf-8
                    fs_encoding: utf-8
                            pid: 1556035
                            cwd: /home/jepler/src/wwvbpy
                           path: /home/jepler/src/wwvbpy
                                 /tmp/pypy3.8-v7.3.6-linux64/lib/pypy3.8
                                 /tmp/pypy3.8-v7.3.6-linux64/lib/pypy3.8/site-packages
                    environment: HOME = /home/jepler
                   command_line: /tmp/pypy3.8-v7.3.6-linux64/lib/pypy3.8/site-packages/coverage/__main__.py debug sys
                sqlite3_version: 2.6.0
         sqlite3_sqlite_version: 3.34.1
             sqlite3_temp_store: 0
        sqlite3_compile_options: COMPILER=gcc-10.2.1 20210110; ENABLE_COLUMN_METADATA; ENABLE_DBSTAT_VTAB
                                 ENABLE_FTS3; ENABLE_FTS3_PARENTHESIS; ENABLE_FTS3_TOKENIZER
                                 ENABLE_FTS4; ENABLE_FTS5; ENABLE_JSON1
                                 ENABLE_LOAD_EXTENSION; ENABLE_PREUPDATE_HOOK; ENABLE_RTREE
                                 ENABLE_SESSION; ENABLE_STMTVTAB; ENABLE_UNLOCK_NOTIFY
                                 ENABLE_UPDATE_DELETE_LIMIT; HAVE_ISNAN; LIKE_DOESNT_MATCH_BLOBS
                                 MAX_SCHEMA_RETRY=25; MAX_VARIABLE_NUMBER=250000; OMIT_LOOKASIDE
                                 SECURE_DELETE; SOUNDEX; TEMP_STORE=1
                                 THREADSAFE=1; USE_URI

1. Clone https://github.com/jepler/wwvbpy and switch to branch pypy38
2. Install requirements with pip install -r requirements_dev.txt (using a virtualenv if desired)
3. Run tests using python3 with make coverage. This will SUCCEED with 100% coverage
4. Install https://downloads.python.org/pypy/pypy3.8-v7.3.6-linux64.tar.bz2 -- untar (e.g., in /tmp), pypy38 -mensurepip, pypy38 -mpip install -r requirements_dev.txt (using the full path to pypy38 as necessary) (this may also require a rust compiler be installed)
5. Run tests using pypy38 with make PYTHON=pypy38. This will FAIL with missing branch coverage in bcd_bits, in particular that "line 703 didn't jump to line 701, because the loop on line 703 didn't complete":

   698 def bcd_bits(n: int) -> Generator[bool, None, None]: 
   699     """Return the bcd representation of n, starting with the least significant bit""" 
   700     while True: 
   701         d = n % 10 
   702         n = n // 10 
   703         for i in (1, 2, 4, 8):                                                          # 703 ↛ 701
   704             yield bool(d & i) 

I'm sorry that I don't have a minimal test case. I attempted to distill this down to a simple program with just def bcd_bits and a driver function, but when I did so 100% branch coverage was reported with pypy3.8. I'm very confident that in actual use, bcd_bits is called with "large" values like 366, with 10 bits consumed from the generator, and of course reassured by the reported complete coverage under a range of other Python versions including pypy3.7.

[nedbat]

Thanks, I think this might be coverage.py not understanding the difference between PyPy3.7 and PyPy3.8. The coverage.py test suite itself has some failures under PyPy3.8 that I have to look at.

[nedbat]

I've written a bug against PyPy3.8: https://foss.heptapod.net/pypy/pypy/-/issues/3577 though I'm not quite sure why that would manifest with the coverage message you are seeing.

[jepler]

Thanks for pursuing this with upstream!

[jepler]

Thank you again for pursuing this, the fix in the newest pypy release also clears up my original problem.