Generators

Let's go back to the book with a bookmark. Picture a reader who reads aloud to you one chapter at a time. Having finished a chapter, the reader puts in a bookmark and falls silent. Ask for more, and the reader opens the book at the bookmark and reads the next chapter, with the notes in the margins still in place.

An ordinary function cannot do that. To give out a series of values, it collects them into a list and returns them all at once. But a list takes up memory in full, even when only the first elements are needed. A function whose body contains the word yield gives out values one at a time instead, like that reader:

def evens(limit):
    for n in range(0, limit, 2):
        yield n


g = evens(7)
print(next(g), next(g))  # 0 2
print(list(g))  # [4, 6]

The call evens(7) does not run the body; it returns a generator — an iterator that computes values on request. Let's go through the example line by line:

1. g = evens(7) creates a generator, but not a single line of the body has run yet.
2. The first next(g) runs the body up to yield n and gets 0.
3. The second next(g) continues from where it stopped and gets 2.
4. list(g) takes the remaining values: [4, 6].

The pause at yield

Each next(g) runs the body up to the next yield, returns its value and pauses the function. Local variables such as n are kept until the next next. The next call continues from the line after yield, not from the start of the function.

Press "Next step" and watch the frame counter. It appears on each next and leaves at yield, while n never resets to 1.

When the body ends, the generator raises StopIteration, and for, list or sum finish without an error. A return in a generator's body also ends it early.

Interviewers like to ask how yield differs from return. The answer: return ends the function for good, while yield gives out a value and pauses the function, keeping its local variables.

One pass

A generator is an iterator, so it gives out its values once. Here the reader analogy ends: a reader can start the book over, while a generator never goes back. Take g = evens(5):

• the first list(g) gives [0, 2, 4];
• the second list(g) gives [];
• list(evens(5)) gives [0, 2, 4] again, because it is a new generator.

A second for over the same g never runs its body, and sum(g) gives 0, as for an empty list. To go through the values again, call the function anew or save the result in a list. The generator expression (n * n for n in nums) from the theme on comprehensions creates the same kind of single-use generator.

Lazy evaluation

A generator computes a value only when it is asked for. This is the same lazy evaluation that a generator expression has:

def countdown(n):
    print("start", end=" ")
    while n > 0:
        yield n
        n -= 1


g = countdown(3)
print("created")  # created
print(next(g))  # start 3

The word start appeared only at the first next, not at the call countdown(3). So a generator can even be endless, say with while True: in its body: values are computed while they are taken. A for loop over such a generator is stopped with break, and the remaining values are never computed.

Common mistake

A generator is asked for its length:

g = evens(7)
print(len(g))  # ✗

TypeError: object of type 'generator' has no len()

A generator does not know how many values it will give out until it reaches the end of its body. If you need the length or indexes, collect the values into a list: len(list(evens(7))) gives 4.

Key points

• A function with yield returns a generator when called and does not run the body.
• Each next runs the body up to the next yield and pauses the generator, keeping its local variables.
• A generator is walked once and computes values only on request.
• A generator has no len: collect the values into a list first.