Lesson 32. Property-Based Testing

Mission link: Example-based tests check the cases you thought of. A property states an invariant and hands the search for counterexamples to a library that is better at it than you are, and it reports the smallest input that breaks the claim.
Primary source: Hypothesis documentation
Prerequisites: Lesson 28, Lesson 30

Warm-up

1. ▢ Lesson 30 parametrised a test over the falsy set. What was still true about the coverage?

2. ▢ truncate(s, n) should never return more than n characters. Which inputs would you write tests for?

Know this

An example test says "for this input, expect that output". A property says "for every input, this must hold":

from hypothesis import given, strategies as st

def truncate(s: str, n: int) -> str:
    return s if len(s) <= n else s[: n - 3] + "..."

@given(st.text(), st.integers(min_value=0, max_value=50))
def test_truncate_respects_limit(s: str, n: int) -> None:
    assert len(truncate(s, n)) <= n

The library generates inputs, runs the body many times, and on failure shrinks the counterexample to the smallest one it can still fail with:

s = '0', n = 0

>       assert len(truncate(s, n)) <= n
E       AssertionError: assert 3 <= 0
E        +  where 3 = len('...')
E        +    where '...' = truncate('0', 0)
E       Failing test case: test_truncate_respects_limit(
E           s='0',
E           n=0,
E       )

The bug is real and the report is minimal: with n below 3, s[: n - 3] slices from the end and the ellipsis alone already exceeds the limit. Nobody writes truncate("0", 0) by hand, and the property found it in under a second.

The failing values are three out of the fifty-one you declared, at one end of the range. Nothing about that stretch is special to look at; it is only special to truncate.

Shrinking is what makes this practical. Without it a failure arrives as a 400-character string of astral-plane characters, and reading it is the whole job.

Strategies

Defaults are deliberately hostile: st.text() produces empty strings, whitespace, combining characters and emoji; st.floats() produces nan and -0.0 unless excluded. That is the point, and if your function genuinely does not accept those, say so in the strategy rather than in the test body.

st.builds and st.from_type are what make this usable on a real domain: annotate a dataclass, from lesson 15, and the strategy comes for free.

The properties worth looking for

Five shapes cover most real uses:

Round trips are the highest-value and the easiest to spot: serialisation, parsing, encoding, compression, currency formatting, URL building. Any pair of functions named to_ and from_ is a property waiting to be written.

The oracle shape is worth naming because it applies to optimisation. When you replace a straightforward implementation with a fast one, keep the slow one in the test file and assert they agree on generated input. That test is stronger than any set of examples and it costs three lines.

Steering the generator

from hypothesis import given, assume, example, settings, strategies as st

@given(st.lists(st.integers(), min_size=1))
@example([0])                                  # always try this one
@settings(max_examples=500)
def test_average_within_bounds(xs: list[int]) -> None:
    assume(len(set(xs)) > 1)                   # skip uninteresting cases
    assert min(xs) <= average(xs) <= max(xs)

• min_size=1 in the strategy is better than assume(xs) in the body, because filtering throws generated cases away and can exhaust the generator.
• @example pins a case you care about, usually one a property already found, so it is checked first every run.
• assume discards a case; use it rarely, and prefer a more precise strategy.
• @settings(max_examples=...) trades runtime for search. The default is a hundred, which is right for a fast property.

Hypothesis also keeps a database of previously failing examples, in .hypothesis/, and replays them first. So a fixed bug stays checked, and a flaky-looking property that failed once yesterday fails immediately today. Add that directory to your ignore file, and know it exists when a failure appears with no code change.

Where it does not fit

• Anything with no expressible invariant. "The invoice PDF looks right" is not a property.
• Slow operations. A hundred examples times a database round trip is a minute per test.
• Exact business rules. "German VAT is 19 per cent" is an example, and parametrisation from lesson 30 is the tool.
• As a replacement for examples. Keep both: examples document intent and read as specification, properties find the inputs nobody imagined.

The honest workflow is example tests for the behaviour you are building, plus one or two properties for the invariants that must never break. Two properties on a serialisation layer are worth more than fifty examples; zero examples on a pricing rule is unreadable.

Practice

1. ▢ Name the property, and write it.

   def to_query_string(params: dict[str, str]) -> str: ...
   def from_query_string(qs: str) -> dict[str, str]: ...
   

@given(st.dictionaries(st.text(min_size=1), st.text()))
def test_query_string_round_trip(params: dict[str, str]) -> None:
    assert from_query_string(to_query_string(params)) == params

2. ▢ Why is the strategy better than the assume?

   @given(st.lists(st.integers()))
   def test_first_is_smallest(xs):
       assume(len(xs) > 0)
       assert sorted(xs)[0] == min(xs)
   

@given(st.lists(st.integers(), min_size=1))

3. ▢ Match each function to the property shape.

   • a) normalise_whitespace(s)
   • b) compress(data) and decompress(data)
   • c) A new binary search replacing a linear scan
   • d) merge(a, b) for two sorted lists

4. ▢ @given(st.floats()) fails immediately on a function that averages numbers. Is that a bug in the code or the test?

st.floats(allow_nan=False, allow_infinity=False, min_value=0, max_value=1e9)

5. ▢ A property fails once in CI and passes on re-run, with no code change. What happened, and what do you do?

6. ▢ A colleague replaces the whole example suite of a parser with three properties. Argue against.

Real-world reps

• [ ] Find a to_/from_ or encode/decode pair in code you own and write the round-trip property. Expect it to fail, and treat the first failure as a specification question rather than a bug report.
• [ ] Write one oracle property for a function you have optimised, keeping the slow version in the test file.
• [ ] Take a function with an invariant you would state in a docstring, such as "never returns more than n items", and turn that sentence into a property.
• [ ] Add @example(...) for every counterexample a property finds, so the fix stays checked.
• [ ] Tomorrow: annotate one dataclass fully and try st.from_type on it, to see how little strategy code a typed domain needs.

Going further

• Hypothesis documentation: the tutorial, then the strategies reference
• What you can generate and how: every built-in strategy, and composing your own
• Settings: max_examples, deadline, derandomize, and the example database
• assume and filtering: when filtering is acceptable and when it exhausts the generator
• Resources

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.