Generator expressions

In the previous lessons, the conveyor put all the tiles into a box at once. But sometimes the box is not needed. Picture a press that does not work ahead: the belt moves one tile forward only when someone at the end takes it. Such a conveyor hands out tiles on request and keeps nothing.

Why would you want such a conveyor? Look at a sum.

The sum function returns the sum of the elements of an iterable. To add up the squares of numbers, you can pass it a list comprehension. But the comprehension first builds the whole list of squares, even though it is needed for only one pass. A generator expression does without the list:

nums = [3, 1, 4]
print(sum([n * n for n in nums]))  # 26
print(sum(n * n for n in nums))  # 26
squares = (n * n for n in nums)
print(list(squares))  # [9, 1, 16]
print(list(squares))  # []

Let's go through it line by line:

1. sum([n * n for n in nums]) builds the list [9, 1, 16] and adds it up, which gives 26.
2. sum(n * n for n in nums) gets the same squares one at a time, without a list, and also prints 26.
3. squares = (n * n for n in nums) computes nothing yet: this line only sets up the conveyor.
4. The first list(squares) takes all the tiles and prints [9, 1, 16].
5. The second list(squares) prints []: the belt is already empty.

A generator expression is written like a list comprehension, but in parentheses. It does not build a list; it hands out values one at a time when they are asked for. Such computing on demand is called lazy evaluation. A list of a million numbers takes megabytes of memory, while a generator expression keeps only the current step of the loop.

Parentheses and printing

If a generator expression is the only argument of a call, a second pair of parentheses is not needed: sum(n * n for n in nums). Inside a call with several arguments, you write the parentheses, as in the line squares = (n * n for n in nums).

print(squares) shows not the values but a string like <generator object <genexpr> at 0x…>, as with enumerate. The list function collects the values into a list.

One pass

You can go through a generator expression only once. When all its values have been handed out, it is empty, and the second list(squares) returned [] without any error. This is where the analogy stops working: a real belt can be loaded again, but an empty generator expression cannot be restarted. For a second pass, write it again or save the values in a list.

Interviewers like to ask this: "How does a generator expression differ from a list comprehension?" The answer has three parts:

• a comprehension builds the whole list at once, while a generator expression hands out values on request;
• a generator expression keeps only the current step, however many values it hands out;
• you go through it once, and it has no indices and no length.

any and all

The any function returns True if at least one element is true, and all returns True if all of them are. With a generator expression, they test a condition for each element:

nums = [3, 1, 4]
print(any(n > 3 for n in nums))  # True
print(all(n > 3 for n in nums))  # False

Both functions stop as soon as the answer is clear—like or and and:

• any stops at the first true value;
• all stops at the first false one.

The remaining values are never computed. Here all needed only the first number: 3 > 3 is false.

min and max with key

min and max accept not only several arguments but also a single iterable. With a generator expression, max(len(w) for w in words) finds the greatest length—the number 6, not the word itself. The keyword argument key gives back the word: it sets which value the elements are compared by.

words = ["pear", "fig", "banana"]
print(max(len(w) for w in words))  # 6
print(min(words), max(words))  # banana pear
print(min(words, key=len), max(words, key=len))  # fig banana

Let's go through the three lines:

1. max(len(w) for w in words) compares lengths and returns the number 6.
2. Without key, strings are compared alphabetically, letter by letter, so min(words) is banana and max(words) is pear.
3. With key=len, the functions compare lengths but return the element itself—the word, not the number.

The function name after key= is written without parentheses: min and max call it themselves, once for each element.

Common mistake

The parentheses are taken for a tuple, and an element is read by index:

nums = [3, 1, 4]
squares = (n * n for n in nums)
print(squares[0])  # ✗

TypeError: 'generator' object is not subscriptable

A generator expression does not store values, so it has neither indices nor a length for len. If you need a list, write the comprehension in square brackets or call list(squares).

Key points

• The generator expression (n * n for n in nums) hands out values one at a time and builds no list.
• A sole argument of a call needs no second pair of parentheses: sum(n * n for n in nums).
• You go through a generator expression once, and it has no indices and no len.
• min(words, key=len) compares lengths but returns the word itself.