Iterators and generators

Every for loop runs the same protocol. for x in thing first calls iter(thing) to get an iterator, then calls next() on it until it raises StopIteration, which the loop swallows and treats as "done".

• An iterable is anything with __iter__: lists, dicts, strings, files, ranges. Each call to iter() on a list gives a new iterator starting from the beginning.
• An iterator has __next__ and keeps a position. Its __iter__ returns itself, so it can be used in a for loop too — but it only moves forwards, and once it has raised StopIteration it stays finished.

1def run():2    nums = [10, 20]3    it = iter(nums)4    steps = [next(it), next(it)]5    try:6        next(it)7    except StopIteration:8        steps.append("StopIteration")9    return {10        "steps": steps,11        "iterator_is_its_own_iter": iter(it) is it,12        "list_gives_fresh_iterators": iter(nums) is not iter(nums),13        "after_exhaustion": list(it),14        "list_again": list(nums),15    }

Generators

A function containing yield is a generator function. Calling it runs none of its body; it returns a generator object, which is an iterator. Each next() runs the body up to the next yield, hands that value out, and freezes the function's local state until the following next(). Falling off the end (or return) raises StopIteration.

def squares_below(limit):
    n = 0
    while n * n < limit:
        yield n * n
        n += 1

list(squares_below(20))   # [0, 1, 4, 9, 16]

That laziness is the point: a generator produces one value at a time, so it can describe a sequence too large to hold in memory, or an infinite one, and the consumer decides how much to take (itertools.islice(gen, 5)). A generator expression is the lazy form of a list comprehension: (x * x for x in nums) instead of [x * x for x in nums]. yield from other yields everything other produces, which keeps generators that combine other iterables short.

1from itertools import islice2 3log = []4 5 6def numbers():7    n = 18    while True:  # infinite, and that is fine9        log.append(f"make {n}")10        yield n11        n += 112 13 14def chain(*iterables):15    for it in iterables:16        yield from it17 18 19def run():20    log.clear()21    gen = numbers()22    log.append("created")23    squares = (n * n for n in gen)24    for sq in islice(squares, 3):25        log.append(f"use {sq}")26    return {"log": log, "chained": list(chain([1, 2], "ab", range(3)))}

One pass only

A generator, like any iterator, can be consumed once. Loop over it a second time and the loop body simply never runs — no error, just nothing. x in gen consumes items up to the match, and len(gen) does not exist. If you need the values twice, keep a list, or call the generator function again for a fresh generator.

1def run():2    squares = (n * n for n in range(5))3    first = sum(squares)4    second = sum(squares)5 6    letters = iter("abcdef")7    found = "c" in letters8    remaining = "".join(letters)9 10    try:11        len(n for n in range(3))12        length = "ok"13    except TypeError as e:14        length = f"TypeError: {e}"15    return {"first_sum": first, "second_sum": second, "found_c": found,16            "left_after_in": remaining, "len": length}

Common mistakes

• Reusing an iterator. The second sum() over the same generator returns 0, and in silently ate the items it searched past.
• Expecting a generator function to do work when it is called. Nothing runs until the first next(); a validation check at the top of the body fires late, at the first iteration.
• Treating a generator like a list: len(), indexing and slicing all raise TypeError. Use list(gen) when you really need all of it.

What interviewers ask

• What is the difference between an iterable and an iterator? An iterable hands out iterators through __iter__; an iterator has __next__, keeps a position and is exhausted after one pass.
• Why use a generator instead of returning a list? Memory and latency: values are produced on demand, the first one is available immediately, and infinite or huge sequences become possible.
• What happens when a generator function is called? It returns a generator object without running the body; next() runs it to the next yield.