Lesson 8. The Iteration Protocol

Mission link: A large share of the confusing bugs in data-handling code are one iterator that was already consumed. The protocol is four lines long, and knowing it turns those bugs into predictions.
Primary source: The Python Standard Library, Iterator Types
Prerequisites: Lesson 3, Lesson 4, Lesson 7

Warm-up

1. ▢ Lesson 4 said d.keys() is a live view rather than a snapshot. What happens if you add a key while looping over it?

2. ▢ Lesson 7 called (n * n for n in range(3)) a generator expression rather than a tuple. Guess what the second sum prints.

   g = (n * n for n in range(3))
   print(sum(g))
   print(sum(g))
   

Know this

for is sugar. This is what it does, exactly:

it = iter(items)                 # 1. ask the object for an iterator
while True:
    try:
        item = next(it)          # 2. ask the iterator for a value
    except StopIteration:        # 3. it says it has none left
        break
    ...                          # 4. the loop body

That gives two distinct roles, and the difference between them is the whole lesson.

An iterable can produce an iterator. It implements __iter__.

An iterator produces values one at a time and remembers where it stopped. It implements __next__, and its __iter__ returns itself.

Every iterator is also an iterable, because its __iter__ hands back itself. That is why for accepts either one without caring. The reverse is not true, and that asymmetry produces every bug below.

An iterator is consumed once, and says nothing about it

pairs = zip([1, 2], "ab")
print(list(pairs))               # [(1, 'a'), (2, 'b')]
print(list(pairs))               # []

Nothing raised. The second list asked for a value, was told there were none, and built an empty list. That silence is the danger: the code looks like it processed the data twice and it processed it once.

A list survives repeated iteration because each for calls iter and gets a fresh iterator with its own position. zip is already the iterator, so there is nothing fresh to get.

The rule that prevents this: if a value has to be traversed twice, it must be a container, not an iterator. Materialise it once, deliberately.

pairs = list(zip([1, 2], "ab"))  # now re-iterable

A file is its own iterator

with open(path) as f:
    line_count = sum(1 for _ in f)
    first = next(f, None)        # None: the file is at its end

There is no second pass without f.seek(0). A function that takes "the lines" and loops over them twice works when it is handed a list and quietly returns nothing on the second pass when it is handed a file. This is the most common form of the bug in real code, because the two callers look identical at the call site.

next with a default, and the sentinel form of iter

next(it) raises StopIteration when exhausted. next(it, default) returns the default instead, which is the readable way to say "the first one, if there is one".

with open(path) as f:
    next(f, None)                # discard the header line
    for line in f:               # continues from line 2
        ...

Partially consuming an iterator and then continuing is not a trick, it is the point of holding one. itertools.islice does the same job for a range of positions.

iter also has a two-argument form that calls something until it returns a sentinel:

for chunk in iter(lambda: stream.read(4096), b""):
    ...

Making your own type iterable

Almost always: return an iterator built from something that already is one.

class Deck:
    def __init__(self, cards):
        self._cards = list(cards)

    def __iter__(self):
        return iter(self._cards)

Deck is now iterable, re-iterable, and works with for, list, sum, in, and comprehensions. Two loops over one deck do not interfere, because each gets its own iterator.

Defining __next__ on Deck itself would make the deck its own iterator, and then two loops over one deck would fight over a single shared position. That is what the file object does, and it is a deliberate choice for a stream, not a default to copy.

One legacy path is worth recognising in old code: a class with __getitem__ taking integers from zero is iterable without __iter__, because iter falls back to calling obj[0], obj[1] and so on until IndexError.

When you genuinely need two passes

Three honest options, in order of preference:

1. Store it: items = list(source). Costs memory, and it is usually the right answer.
2. Recompute it: call the function that produced the iterator again. Right when the source is cheap or huge.
3. itertools.tee: buffers whatever one branch has seen and the other has not. Right for two passes that advance roughly together, wrong as a way to avoid deciding.

Practice

1. ▢ Predict both lines.

   nums = map(int, ["1", "2", "3"])
   print(max(nums))
   print(sum(nums))
   

2. ▢ This function is called twice: once with a list of lines, once with an open file. Describe both outcomes.

   def summarise(lines):
       total = sum(len(line) for line in lines)
       longest = max((line for line in lines), key=len)
       return total, longest
   

3. ▢ Which of these can be looped over twice with the same result? For each no, say what it is instead.

   • a) sorted(nums)
   • b) reversed(nums)
   • c) nums.keys() for a dict nums
   • d) (n for n in nums)
   • e) range(len(nums))

4. ▢ Write first_matching(items, predicate), returning the first item that satisfies predicate or None. Do it without a for statement, and make it stop at the first match.

def first_matching(items, predicate):
    return next((item for item in items if predicate(item)), None)

5. ▢ Why does the second loop print nothing, and what are the two ways to fix it?

   evens = filter(lambda n: n % 2 == 0, range(10))
   for n in evens:
       print(n)
   for n in evens:
       print("again", n)
   

6. ▢ A colleague makes their Playlist class its own iterator, so __next__ lives on Playlist and tracks a self._position. Name a specific way this breaks.

Real-world reps

• [ ] Grep code you know for zip(, map(, filter( and enumerate(. For each one, ask whether the result is traversed once. Anything traversed twice is either already a bug or one edit away from being one.
• [ ] Take a function that accepts "a list of things" and loops over the argument twice. Hand it a generator expression and watch what it returns. Then fix it, and decide whether the fix belongs in the function or in its documented signature.
• [ ] Add __iter__ to a class you own that currently exposes a .items list, and remove the callers reaching into that attribute.
• [ ] Tomorrow: read a header line off a file with next(f, None) and process the rest with a plain for, instead of an index check inside the loop.

Going further

• Iterator Types: the protocol stated in full, including the requirement that __iter__ returns self
• Glossary: iterable, iterator, sequence: the canonical one-line definitions
• itertools: islice, tee, chain and the rest, all of which take and return iterators
• The for statement: the exact desugaring, including what break does to the iterator
• Resources

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.