Lesson 28. What a Test Asserts

Mission link: The mission asks for tests that fail informatively. That is a property of how a test is written, not of the framework, and it is the difference between a red build you can read and one you have to reproduce locally.
Primary source: pytest documentation, Get Started
Prerequisites: Lesson 10, Lesson 13

Warm-up

1. ▢ Lesson 10: why is assert amount > 0 the wrong way to validate user input?

2. ▢ What is the single most useful thing a failing test can tell you?

Know this

A test is a function whose name starts with test_, in a file named test_*.py, and it asserts with the plain assert statement:

def total(items):
    return sum(i["amount"] for i in items)

def test_totals_two_items():
    items = [{"amount": 10}, {"amount": 2}]
    assert total(items) == 13

    def test_totals_two_items():
        items = [{"amount": 10}, {"amount": 2}]
>       assert total(items) == 13
E       AssertionError: assert 12 == 13
E        +  where 12 = total([{'amount': 10}, {'amount': 2}])

No assertEqual, no message argument. pytest rewrites the assertion at import time to report both sides and how each was computed, and it does the same for containers:

E       assert [1, 2, 3] == [1, 2, 4]
E         At index 2 diff: 3 != 4

E       AssertionError: assert {'a': 1, 'b': 2} == {'a': 1, 'b': 3}
E         Omitting 1 identical items, use -vv to show
E         Differing items:
E         {'b': 2} != {'b': 3}

E       AssertionError: assert 'hello world' == 'hello werld'
E         - hello werld
E         ?        ^
E         + hello world
E         ?        ^

This is the practical reason for __repr__ from lesson 24: every one of those messages is repr output. A class without one turns the third message into two object addresses.

The shape of a test

def test_rejects_negative_amount():
    order = Order(id=1, amount=Decimal("10.00"))     # arrange

    with pytest.raises(ValueError, match="negative"):
        order.apply_discount(Decimal("-5.00"))       # act, and assert

Three properties, in order of how much they matter.

One behaviour per test. Not one assertion, which is a rule people cite and which forces awkward tests: several assertions about one behaviour are fine. What is not fine is a test that exercises three behaviours, because when it fails you learn only that one of the three broke.

A name that states the behaviour. test_rejects_negative_amount tells you what broke from the summary line alone. test_order_2 requires opening the file, and test_apply_discount names the method rather than the claim.

No logic. A test with an if, a loop, or a computed expected value has a second implementation of the thing under test inside it, and now two things can be wrong. Write the expected value as a literal, even when that means repetition. Parametrisation, in lesson 30, is how to avoid the loop.

The assertions worth knowing

assert result == expected                    # the default, and usually enough
assert result is None                        # identity, from lesson 5
assert 0.1 + 0.2 == pytest.approx(0.3)       # floats, from lesson 14
assert "negative" in str(exc.value)          # part of a message

with pytest.raises(ValueError, match="negative"):
    ...                                      # match is a regex against str(exc)

with pytest.raises(ValueError) as exc_info:
    ...
assert exc_info.value.field == "amount"      # inspect the exception object

pytest.raises without an argument to narrow it is a weak test: it passes when the code raises the right exception for the wrong reason, and it passes when a typo raises NameError. Pass match=, or assert on the exception object.

Running and selecting

pytest                          # discover and run everything
pytest tests/test_orders.py     # one file
pytest -k "discount"            # tests whose name matches
pytest -x                       # stop at the first failure
pytest --lf                     # only what failed last time
pytest -q                       # one line per file instead of per test
pytest -vv                      # full diffs, no truncation

--lf and -x together are the debugging loop. -vv is the answer to "use -v to get more diff" in the output above.

Where tests live

shop/
├── src/shop/orders.py
└── tests/
    ├── conftest.py             # shared fixtures, lesson 29
    └── test_orders.py

Tests outside the package, importing it as an installed package, which is lesson 19's editable install. That way the tests exercise what users get, and tests/ is not shipped in the wheel.

Two mechanical notes that cause most first-day confusion. Test files need unique names when there is no __init__.py in tests/, because otherwise two test_utils.py collide as modules. And a test that needs the package importable needs the package installed, or pythonpath = ["src"] in the pytest configuration, which is the shortcut.

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-ra --strict-markers"

-ra prints a summary of everything that was not a plain pass, which is how skipped and expected-failure tests stop being invisible. --strict-markers turns a misspelled marker into an error rather than a silently ignored one.

What deserves a test

The mission's answer, in one sentence: anything whose failure you would want to hear about from a machine rather than a user.

The regression test is the highest-value test there is, and it has an unfair advantage: you know it fails before you fix anything. Write it first, watch it fail, then fix.

Practice

1. ▢ This test passes. Explain why it is nearly worthless.

   def test_apply_discount():
       order = Order(id=1, amount=Decimal("100"))
       with pytest.raises(Exception):
           order.apply_discount(Decimal("-5"))
   

with pytest.raises(ValueError, match="must not be negative"):
    order.apply_discount(Decimal("-5"))

2. ▢ Split this into the right number of tests, and name them.

   def test_order():
       order = Order(id=1, amount=Decimal("100"), country="GB")
       assert order.amount == Decimal("100")
       order.apply_discount(Decimal("10"))
       assert order.amount == Decimal("90")
       assert order.vat() == Decimal("18")
       with pytest.raises(ValueError):
           order.apply_discount(Decimal("-1"))
   

def test_stores_the_amount_it_was_given(): ...
def test_discount_reduces_the_amount(): ...
def test_vat_is_twenty_percent_for_gb(): ...
def test_rejects_a_negative_discount(): ...

3. ▢ Which of these tests has logic in it, and why does that matter?

   • a) assert total([{"amount": 10}, {"amount": 2}]) == 12
   • b) assert total(items) == sum(i["amount"] for i in items)
   • c) for n in (0, 1, 5): assert double(n) == n * 2
   • d) assert format_money(Decimal("1234.5")) == "1,234.50"

4. ▢ Why does assert result == expected beat assert result == expected, "totals should match"?

5. ▢ A test suite has 400 tests, all passing, and a production bug ships in a function with three branches. Where do you look first?

6. ▢ Your Order class has no __repr__ and a test comparing two orders fails. What does the output look like, and what is the one-line fix?

>       assert Order(1) == Order(2)
E       assert <shop.orders.Order object at 0x1094757f0> == <shop.orders.Order object at 0x1094982d0>
E        +  where <shop.orders.Order object at 0x1094757f0> = Order(1)
E        +  and   <shop.orders.Order object at 0x1094982d0> = Order(2)

Real-world reps

• [ ] Take the last bug you fixed and write the test after the fact. Then revert the fix, confirm the test fails, and reapply it. That loop is the only proof a test detects anything.
• [ ] Find a test in your suite named after a method rather than a behaviour, and rename it. Read the summary line afterwards.
• [ ] Grep for pytest.raises(Exception) and raises(Exception) and narrow each one, adding match=.
• [ ] Find a test with a loop or an if and convert it to literals. Note whether it was reimplementing the code under test.
• [ ] Tomorrow: add -ra --strict-markers to your pytest configuration and read the summary. Skipped tests you had forgotten about are the usual discovery.

Going further

• Get Started: installation, discovery rules, and the first assertion
• How to write and report assertions: the rewriting, pytest.raises, and approx
• Good Integration Practices: where tests live, and why the src layout matters here
• Configuration: the [tool.pytest.ini_options] table
• Resources

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.