Iterators

Picture reading a book in the evenings. When you stop, you put a bookmark between the pages. The next evening you open the book at the bookmark and read on, not from the first page.

A for loop walks over lists, strings, dicts, range and files, and all of them are built differently. What lets it take values one at a time is an iterator — an object that remembers where the walk stopped. An iterator is like a bookmark: the book itself is the list, and the bookmark keeps only the place where you stopped.

The function iter gives an iterator, and the function next gives the next value:

nums = [10, 20]
it = iter(nums)
print(next(it))  # 10
print(next(it))  # 20
print(next(it, "end"))  # end

Let's go through the example line by line:

1. iter(nums) puts a bookmark before the first element of the list and returns the iterator it.
2. The first next(it) returns 10 and moves the bookmark by one element.
3. The second next(it) returns 20: the bookmark remembers where it stopped.
4. The elements have run out, and next(it, "end") returns its second argument.

Without the second argument, next(it) on a used-up iterator raises the exception StopIteration. An iterator does not go back and does not know its length: it has only the next element. This is where the analogy ends: you can move a paper bookmark back to the start, while an iterator only moves forward.

What for does

The loop for ch in "ab": takes the same steps as this program:

it = iter("ab")
while True:
    try:
        ch = next(it)
    except StopIteration:
        break
    print(ch, end=".")  # a.b.

Let's see what the for loop does here:

1. It calls iter once, before the first iteration.
2. Before each iteration, it calls next and stores the result in the loop variable.
3. When next raises StopIteration, the loop catches the exception and finishes.

For the loop, StopIteration is not an error but the end signal. A dict's iterator gives out keys, so for k in d walks the keys of the dict d.

Iterable and iterator

Interviewers like to ask this: "How does an iterable differ from an iterator?" Let's look at both ideas.

An iterable is the book

An iterable is anything iter can make an iterator from: a list, a string, a dict, a range. It is the book. You can read it any number of times, because each call to iter puts a new bookmark at the start: iter(nums) is iter(nums) gives False.

An iterator is the bookmark

An iterator is the bookmark itself, and it is walked once: once used up, it stays empty. zip and enumerate return iterators, not lists, and a file object is an iterator too. Take pairs = zip("ab", [1, 2]):

• the first list(pairs) gives [('a', 1), ('b', 2)];
• the second list(pairs) gives [], because the bookmark is already at the end.

An iterator is iterable too: iter(pairs) returns pairs itself. So a for over an iterator continues from the place where the walk stopped.

Common mistake

next is called right on a list:

nums = [10, 20]
print(next(nums))  # ✗

TypeError: 'list' object is not an iterator

A list is an iterable but not an iterator: it keeps no walk position. A string, a dict and a range give the same TypeError: all of them are iterables but not iterators. Get an iterator first: next(iter(nums)) gives 10.

Key points

• iter gives an iterator, and next gives its next element; at the end, next raises StopIteration.
• next(it, default) returns default instead of raising.
• for calls iter and next itself and finishes on StopIteration.
• An iterable can be walked many times, an iterator only once.