enhance unittest to show test name and docstring on one line

[ethanfurman]

BPO	47133
Nosy	@gpshead, @stevendaprano, @ethanfurman, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ethanfurman'
closed_at = None
created_at = <Date 2022-03-26.20:56:32.850>
labels = ['easy', 'type-feature', 'tests', '3.11']
title = 'enhance unittest to show test name and docstring on one line'
updated_at = <Date 2022-04-03.20:00:29.618>
user = 'https://github.com/ethanfurman'

bugs.python.org fields:

activity = <Date 2022-04-03.20:00:29.618>
actor = 'gregory.p.smith'
assignee = 'ethan.furman'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2022-03-26.20:56:32.850>
creator = 'ethan.furman'
dependencies = []
files = []
hgrepos = []
issue_num = 47133
keywords = ['easy']
message_count = 9.0
messages = ['416090', '416094', '416097', '416099', '416100', '416103', '416105', '416138', '416642']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'steven.daprano', 'ethan.furman', 'serhiy.storchaka', 'itay.yeshaya']
pr_nums = []
priority = 'low'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue47133'
versions = ['Python 3.11']

[ethanfurman]

When running unittest with the -v flag, if a test has errors, and has a docstring, the test name is shown on one line, and the docstring is shown on the next line -- and the ERROR word is shown with the docstring.

What this means is that:

• the test name is easily missed by anyone who isn't aware of that (I only recently found out when someone else pointed it out to me)
• grepping for ERROR doesn't reveal the test name, necessitating a search through the testing code to find the test that failed

There are two possible solutions, and the selection of which one should be discussed on python-dev:

• print the name and docstring, and ERROR, all on one line; or
• provide a command-line switch to select that behavior

---

For the experienced developers who see this: please leave the issue for one of the new developers subscribed to core-mentorship. Thank you.

[itayyeshaya]

I would like to work on this.

[serhiy-storchaka]

bpo-14265, bpo-46126.

[stevendaprano]

I don't think this change of behaviour should be accepted without discussion, and I'm not sure why you think Python-Dev is the right place, rather than here.

Messing around with unittest and docstrings has already caused issues in the past:

https://docs.python.org/3/library/unittest.html#unittest.TestCase.shortDescription

so perhaps we shouldn't break what's not broken?

And frankly, I don't see why the current behaviour is a problem, although maybe that's because the way I use unittest is different from the way you use it. Grepping the output for errors? Why run verbose mode if you aren't going to read the whole output? Seems very odd.

Ethan says: "the test name is easily missed by anyone who isn't aware of that"

Surely the solution to that is education, not changing unittest? Now you are aware of it, and the test name is no longer "easily missed".

If it ever was. To me, it is plain as day. Here is some typical output:

[steve ~]$ python3.10 -m unittest -v mathlib/testing/test_utils.py
test_broken_doc (mathlib.testing.test_utils.TestMinmax)
This is a docstring ... ERROR
test_broken_nodoc (mathlib.testing.test_utils.TestMinmax) ... ERROR
test_empty_iterable (mathlib.testing.test_utils.TestMinmax) ... ok
[additional passing tests ommitted for brevity]

======================================================================
ERROR: test_broken_doc (mathlib.testing.test_utils.TestMinmax)
This is a docstring
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/steve/Desktop/mathlib/testing/test_utils.py", line 20, in test_broken_doc
    self.assertEqual(1, x)
NameError: name 'x' is not defined

======================================================================
ERROR: test_broken_nodoc (mathlib.testing.test_utils.TestMinmax)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/steve/Desktop/mathlib/testing/test_utils.py", line 24, in test_broken_nodoc
    self.assertEqual(1, x)
NameError: name 'x' is not defined

---

Ran 10 tests in 0.004s

FAILED (errors=2)

Errors are listed twice, once in the summary, and the second time in the details section showing the traceback. Note that if you just grep for ERROR you get the result you want:

[steve ~]$ python3.10 -m unittest -v mathlib/testing/test_utils.py |& grep ERROR
This is a docstring ... ERROR
test_broken_nodoc (mathlib.testing.test_utils.TestMinmax) ... ERROR
ERROR: test_broken_doc (mathlib.testing.test_utils.TestMinmax)
ERROR: test_broken_nodoc (mathlib.testing.test_utils.TestMinmax)

So I am having trouble seeing what the problem here is.

[ethanfurman]

I proposed a discussion to python-dev to increase the odds that folks with an interest could chime in. Not everyone follows the new-bugs list.

I find having the output on two lines counter-intuitive -- I was expecting one line per test, and indeed my experience seemed to support that (out of hundreds of tests showing on one line, it's easy to miss that the rare verbose error is on two). And I'm not the only one, since another developer also thought there was a problem and spent most of bpo-46126 "fixing" it.

Another reason for posting to python-dev is that bpo-46126 had been active for three months and had a PR merged before another dev mentioned that verbose tests with docstrings just printed on two lines instead of one.

Thank you for pointing out the possible disruption caused by unittest output changes.

[stevendaprano]

I have no problem with pinging Python-Dev, or any other forum, asking
for interested parties. I just don't think we should be intentionally
spliting the discussion more than it's going to get split organically.
(If this is interesting to people, they will discuss it on other forums
no matter what we say or do.)

Aside from the conservative position "If it ain't broke, don't break
it", also known as "Chesterton's Fence":

https://matt-rickard.com/chestertons-fence/

I'm not specifically opposed to this change. Nor am I in favour.

Why do we print the first line of the doctests in the output of failing
tests? If we understood the rationale for why we do that, it might give
some insight into the consequences of removing it. If we don't know why
it was shown in the first place, maybe we shouldn't remove it until we
have some idea.

I am -1 on another option flag unless there is no other choice. Each
boolean flag option doubles the number of tests of unitttest itself
needed for full coverage.

Another option might be to leave the docstring line alone, and just
*add* the 'ERROR' to the method name line:

# current output
test_broken_doc (mathlib.testing.test_utils.TestMinmax)
This is a docstring ... ERROR

# enhancement
test_broken_doc (mathlib.testing.test_utils.TestMinmax) ERROR
This is a docstring ... ERROR

That seems to me to be less risky than trying to cram them all on one line.

test_broken_doc (mathlib.testing.test_utils.TestMinmax) This is a docstring ... ERROR

By the way, I presume we do the same thing for FAIL lines, which have
the same behaviour as ERROR lines:

test_fail_doc (mathlib.testing.test_utils.TestMinmax)
This is a docstring ... FAIL

test_fail_nodoc (mathlib.testing.test_utils.TestMinmax) ... FAIL

[serhiy-storchaka]

This is already the third (at least) issue asking for such feature. Many people have problems with the current output. I would prefer to have a full qualified name of the test which I can copy by a double click and insert in a command that reruns specified tests. And full path to the test's source which I can copy to run in editor. pytest output looks more user-friendly to me.

But it may break other's test output parsers so we perhaps need an option for this. Options for unittest and libregrtest. I am not sure that it is a task for an average new contributor.

[stevendaprano]

Rather than an option for this, it would be cleaner to deprecate the
current output in 3.11, and fix it in 3.12 or 3.13.

Otherwise we have to maintain the old (bad?) output and the new output
both forever.

[gpshead]

A more fundamental issue behind this: unittest's plain text verbose output is not intended to be parsable and consumable by machines. It is 22 years old and was intended for use in 80 column wide terminals as a progress report while watching it run as much it is a final result. Consider that the legacy use case or format if you wish.

Allowing an option for output to one line per test with description could be more likely to require horizontal scrolling or reading past a line wrap in CI setups where text is scrolling regions embedded in scrolling regions. Yuck. BUT not necessarily bad. Just different.

The ideas expressed in this thread of making a more clear text format with the important PASS/FAIL/ERROR/SKIP status up front and a very clear pastable test module.class.test_name seem wise.

I expect most Python projects with nicer test output formatting desires have simply adopted pytest and never looked back.

---

Other python unittest derivative concepts to get clear output of results:

We add a reliably machine parsable XML report of test outcomes in absl-py https://github.com/abseil/abseil-py/tree/main/absl/testing. It's IMNSHO a rather annoying implementation - we wish unittest made plugging in additional output formats easier. We use that generated XML format for results from unittest systems in all of our languages at work, it is picked up and parsed by our internal CI system and used to display nicely formatted results when available.

I assume pytest also provides machine parsable reporting capabilities but I haven't looked.

[Rajendra1308]

Hey @ethanfurman I am interested in working on this issue, I am new to opensource can you please explain where the changes have to be made. Also what exactly is needed to be done.