test_doctest cannot be run directly: let's remove this option

[sobolevn]

Bug report

When running ./python.exe Lib/test/test_doctest.py you get:

» ./python.exe Lib/test/test_doctest.py   
.....................................F................FF.........
======================================================================
FAIL: basics (__main__.test_DocTestFinder)
Doctest: __main__.test_DocTestFinder.basics
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/doctest.py", line 2353, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for __main__.test_DocTestFinder.basics
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 445, in basics

----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 469, in __main__.test_DocTestFinder.basics
Failed example:
    print(tests)  # doctest: +ELLIPSIS
Expected:
    [<DocTest sample_func from test_doctest.py:32 (1 example)>]
Got:
    [<DocTest sample_func from /Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py:32 (1 example)>]
----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 576, in __main__.test_DocTestFinder.basics
Failed example:
    for t in tests:
        print('%2s  %s' % (len(t.examples), t.name))
Expected:
     1  some_module
     3  some_module.SampleClass
     3  some_module.SampleClass.NestedClass
     1  some_module.SampleClass.NestedClass.__init__
     1  some_module.SampleClass.__init__
     1  some_module.SampleClass.a_cached_property
     2  some_module.SampleClass.a_classmethod
     1  some_module.SampleClass.a_classmethod_property
     1  some_module.SampleClass.a_property
     1  some_module.SampleClass.a_staticmethod
     1  some_module.SampleClass.double
     1  some_module.SampleClass.get
     1  some_module.__test__.c
     2  some_module.__test__.d
     1  some_module.sample_func
Got:
     1  some_module
     1  some_module.__test__.c
     2  some_module.__test__.d


======================================================================
FAIL: test_pdb_set_trace (__main__)
Doctest: __main__.test_pdb_set_trace
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/doctest.py", line 2353, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for __main__.test_pdb_set_trace
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 1984, in test_pdb_set_trace

----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 2043, in __main__.test_pdb_set_trace
Failed example:
    try:
        runner.run(test)
    finally:
        sys.stdin = real_stdin
Expected:
    --Return--
    > <doctest test.test_doctest.test_pdb_set_trace[7]>(3)calls_set_trace()->None
    -> import pdb; pdb.set_trace()
    (Pdb) print(y)
    2
    (Pdb) up
    > <doctest foo-bar@baz[1]>(1)<module>()
    -> calls_set_trace()
    (Pdb) print(x)
    1
    (Pdb) continue
    TestResults(failed=0, attempted=2)
Got:
    --Return--
    > <doctest __main__.test_pdb_set_trace[7]>(3)calls_set_trace()->None
    -> import pdb; pdb.set_trace()
    (Pdb) print(y)
    2
    (Pdb) up
    > <doctest foo-bar@baz[1]>(1)<module>()
    -> calls_set_trace()
    (Pdb) print(x)
    1
    (Pdb) continue
    TestResults(failed=0, attempted=2)


======================================================================
FAIL: test_pdb_set_trace_nested (__main__)
Doctest: __main__.test_pdb_set_trace_nested
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/doctest.py", line 2353, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for __main__.test_pdb_set_trace_nested
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 2118, in test_pdb_set_trace_nested

----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython/Lib/test/test_doctest.py", line 2154, in __main__.test_pdb_set_trace_nested
Failed example:
    try:
        runner.run(test)
    finally:
        sys.stdin = real_stdin
    # doctest: +REPORT_NDIFF
Differences (ndiff with -expected +actual):
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(5)calls_set_trace()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(5)calls_set_trace()
    ?             ^^^^^^^
      -> self.f1()
      (Pdb) print(y)
      1
      (Pdb) step
      --Call--
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(7)f1()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(7)f1()
    ?             ^^^^^^^
      -> def f1(self):
      (Pdb) step
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(8)f1()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(8)f1()
    ?             ^^^^^^^
      -> x = 1
      (Pdb) step
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(9)f1()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(9)f1()
    ?             ^^^^^^^
      -> self.f2()
      (Pdb) step
      --Call--
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(11)f2()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(11)f2()
    ?             ^^^^^^^
      -> def f2(self):
      (Pdb) step
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(12)f2()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(12)f2()
    ?             ^^^^^^^
      -> z = 1
      (Pdb) step
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(13)f2()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(13)f2()
    ?             ^^^^^^^
      -> z = 2
      (Pdb) print(z)
      1
      (Pdb) up
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(9)f1()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(9)f1()
    ?             ^^^^^^^
      -> self.f2()
      (Pdb) print(x)
      1
      (Pdb) up
    - > <doctest test.test_doctest.test_pdb_set_trace_nested[0]>(5)calls_set_trace()
    ?            --------- ^^^^^^^
    + > <doctest __main__.test_pdb_set_trace_nested[0]>(5)calls_set_trace()
    ?             ^^^^^^^
      -> self.f1()
      (Pdb) print(y)
      1
      (Pdb) up
      > <doctest foo-bar@baz[1]>(1)<module>()
      -> calls_set_trace()
      (Pdb) print(foo)
      *** NameError: name 'foo' is not defined
      (Pdb) continue
      TestResults(failed=0, attempted=2)


----------------------------------------------------------------------
Ran 65 tests in 0.942s

FAILED (failures=3)

These issues are not easy to fix. Moreover, since they are literally written in docstrings, it is not very readable.

The second problem is that -c option is claimed to be supported:

cpython/Lib/test/test_doctest.py

Lines 3365 to 3380 in 78e6d72

	def test_coverage(coverdir):
	trace = import_helper.import_module('trace')
	tracer = trace.Trace(ignoredirs=[sys.base_prefix, sys.base_exec_prefix,],
	trace=0, count=1)
	tracer.run('test_main()')
	r = tracer.results()
	print('Writing coverage results...')
	r.write_results(show_missing=True, summary=True,
	coverdir=coverdir)
	if __name__ == '__main__':
	if '-c' in sys.argv:
	test_coverage('/tmp/doctest.cover')
	else:
	unittest.main()

However:

• test_main is not defined, so it fails
• When changed to unittest.main() (with all of the other test failures fixed), it still does not work on macos sonoma, no file is created, nothing

I propose to:

• raise an exception that running test_doctest directly is not supported
• remove test_coverage

Linked PRs

• gh-111348: Fix direct invocation of test_doctest; remove test_doctest.test_coverage #111349
• [3.12] gh-111348: Fix direct invocation of test_doctest; remove test_doctest.test_coverage (GH-111349) #111359
• [3.11] gh-111348: Fix direct invocation of test_doctest; remove test_doctest.test_coverage (GH-111349) #111360

[AlexWaygood]

Ran 65 tests in 0.942s

FAILED (failures=3)

Same for me on Windows.

[Eclips4]

Related: #104057

[terryjreedy]

Testing doctest with doctest is a bold move, but it means that if a test fails, we don't know if the bug is in doctest or test_doctest unless tests pass, we edit either one but not both, and then tests fail. In this case, it is unclear if running the tests this way ever worked, and if so, what change made it fail.

When a doctest fails because the expected string is overly specific (here, when name is 'main'), then it seems likely to me that the test is buggy and should be fixed. If a specific test is too hard to fix, or inapplicable, it could be skipped.

From #104057, it seems there are several reasons for failure; some seem easy to fix, some less so. We have not collectively settled on a policy of when to disable direct running.

Unless the coverage module cannot work with test_doctest, I do not see any reason to have custom code in just this file.

[terryjreedy]

I ran coverage: 84% for me. Perhap 1% is the function called '_test' although it looks like it should be be renamed 'main' since it enables running doctest on files from a command line (and should be tested). Note: the obvious way to me to get coverage is to run a test directly. However, coverage is written so that passing ".../test/regrtest.py doctest" instead of ".../test/doctest.py" does work.

I might look at the test failures later today.

[sobolevn]

@terryjreedy We actually fixed failures in the PR (by "we" I mean @skirpichev 😆).
Do you run the coverage on win?

[terryjreedy]

Yes.

[skirpichev]

I ran coverage: 84% for me.

I assume it's for coveragepy (and with --branch). With --coverage I got: 1099 69% doctest (/home/sk/src/cpython/Lib/doctest.py).

[terryjreedy]

Package is now called just 'coverage'. https://pypi.org/project/coverage/ Now 7.3.2, I have 7.2.6
I added "[run]\nbranch = True" to my .coveragerc.
"3x\Lib\doctest.py: 84% 987 statements, 851 run 136 missing 4 excluded 65 partial"
I ran it with main, which is not officially supported.

EDIT: 987 appears to be logical rather than physical lines. I noticed that logical lines are only marked on the first physical line. Not sure if this was always the case.

[skirpichev]

I ran it with main, which is not officially supported.

I think it is: https://devguide.python.org/testing/coverage/#install-coverage

BTW, I think this issue can be closed. But there could be a different one: for improving test coverage of doctest.py. Some public functions aren't tested at all (e.g. run_docstring_examples()).

[AlexWaygood]

BTW, I think this issue can be closed. But there could be a different one: for improving test coverage of doctest.py

I agree, we've fixed the bug that this issue was about. Let's continue the discussion about improving coverage in a separate issue 👍