regrtest.py -T broken

[doerwalter]

BPO	10541
Nosy	@doerwalter, @brettcannon, @abalkin, @pitrou, @vstinner
Files	build.log: shell log build2.log: shell log 2 issue10541.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = <Date 2014-06-29.21:57:31.368>
created_at = <Date 2010-11-26.16:02:12.971>
labels = ['type-bug', 'tests']
title = 'regrtest.py -T broken'
updated_at = <Date 2014-06-30.03:35:38.528>
user = 'https://github.com/doerwalter'

bugs.python.org fields:

activity = <Date 2014-06-30.03:35:38.528>
actor = 'berker.peksag'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2014-06-29.21:57:31.368>
closer = 'belopolsky'
components = ['Tests']
creation = <Date 2010-11-26.16:02:12.971>
creator = 'doerwalter'
dependencies = []
files = ['19824', '19871', '20044']
hgrepos = []
issue_num = 10541
keywords = ['patch']
message_count = 10.0
messages = ['122463', '122508', '122544', '122545', '122810', '123952', '123990', '126864', '179701', '221906']
nosy_count = 6.0
nosy_names = ['doerwalter', 'brett.cannon', 'belopolsky', 'pitrou', 'vstinner', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue10541'
versions = ['Python 3.5']

[doerwalter]

Running regrtest.py with coverage option seems to be broken for the py3k branch at the moment. Run the following commands on the shell:

wget http://svn.python.org/snapshots/python3k.tar.bz2
tar xjf python3k.tar.bz2
cd python
./configure --enable-unicode=ucs4 --with-pydebug
make coverage
./python.exe Lib/test/regrtest.py -T -N test_urllib

This gives the following output:

[1/1] test_urllib
Not printing coverage data for 'Lib/test/regrtest.py': [Errno 2] No such file or directory: 'Lib/test/regrtest.py'
Traceback (most recent call last):
  File "Lib/test/regrtest.py", line 1502, in <module>
    main()
  File "Lib/test/regrtest.py", line 698, in main
    r.write_results(show_missing=True, summary=True, coverdir=coverdir)
  File "/Users/walter/x/pybug/python/Lib/trace.py", line 331, in write_results
    with open(filename, 'rb') as fp:
IOError: [Errno 2] No such file or directory: 'Lib/test/regrtest.py'
[123146 refs]

I'm testing on Mac OS X 10.6.5.

Attached is the complete log of the shell session.

This bug might be related to bpo-10329, as the failing line was introduced in r86303.

[abalkin]

Apparently something in the test changes the cwd.

Both of the following invocations work:

$ ./python.exe `pwd`/Lib/test/regrtest.py -T -N test_urllib
$ ./python.exe -m test.regrtest -T -N test_urllib

I would suggest changing the coverage target in the Makefile.

[pitrou]

make coverage is fine, you just have to use the "-m test.regrtest" form when running the tests.

[abalkin]

I would like to investigate this some more. In theory, regrtest should restore cwd before coverage results are written.

[doerwalter]

OK, I reran the test with::

./python -mtest.regrtest -T -N test_urllib

and this does indeed produce coverage files (for _abcoll, _weakrefset, abc, base64, codecs, collections, contextlib, functools, genericpath, hashlib, locale, mimetypes, os, posixpath, quopri, random, re, sre_compile, sre_parse, ssl, stat, tempfile, textwrap, trace, uu, warnings).

However running the complete test suite via::

./python -mtest.regrtest -T -N -uurlfetch,largefile,network,decimal

fails with::

Not printing coverage data for '/tmp/tmp0fdr9o/t4/sub/subsub/__init__.py': [Errno 2] No such file or directory: '/tmp/tmp0fdr9o/t4/sub/subsub/__init__.py'
Traceback (most recent call last):
  File "/home/coverage/python/Lib/runpy.py", line 160, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/coverage/python/Lib/runpy.py", line 73, in _run_code
    exec(code, run_globals)
  File "/home/coverage/python/Lib/test/regrtest.py", line 1502, in <module>
    main()
  File "/home/coverage/python/Lib/test/regrtest.py", line 698, in main
    r.write_results(show_missing=True, summary=True, coverdir=coverdir)
  File "/home/coverage/python/Lib/trace.py", line 331, in write_results
    with open(filename, 'rb') as fp:
IOError: [Errno 2] No such file or directory: '/tmp/tmp0fdr9o/t4/sub/subsub/__init__.py'
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' encoding='ANSI_X3.4-1968'>

(attached is the complete output of running the test suite (build2.log).)

[abalkin]

Here is a simpler invocation that produces a similar error:

$ ./python.exe -m test.regrtest -T  test_trace test_pkg
...

IOError: [Errno 2] No such file or directory: '/var/folders/qs/qsqFUI2xFUKG+9CTf4z7pU+++TI/-Tmp-/tmpy1iyp7/t4/sub/init.py'

[abalkin]

I am attaching a patch that fixes write_results() and makes test_trace tests restore the tracefunc after they run. This fixes generation off the coverage files, but many tests still fail when traced.

[brettcannon]

At least the test_trace fix for preventing the complete destruction of any preset trace should be looked at and possibly applied separately (prevents coverage.py from covering the entire test suite as well). Should probably add a test to regrtest to make sure that the trace function is not mutated after a test file runs.

[brettcannon]

Alexander's patch LGTM

[python-dev]

New changeset 10cf594ace4b by Alexander Belopolsky in branch 'default':
Fixes bpo-10541: regrtest -T is broken
http://hg.python.org/cpython/rev/10cf594ace4b