The Anatomy of a for Loop

What happens behind the scenes when you run a Python `for` loop? How complex can it be?

Jun 27, 2023

Learning to use the for loop is one of the first steps in everyone's Python journey. Early on, you learn how to loop using range() and how to loop through a list. A bit later, you discover you can loop through strings, tuples, and other iterable data types. You even learn the trick of looping through a dictionary using its items() method.

But what's really happening behind the scenes in a for loop?

Let's look inside the for loop and make sense of the "iter-soup" of terms: iterate, iterable, iterator, iter(), and __iter__().

I will not describe and explain how to use the for loop in this post. If you're looking for a "for loop primer", you can read Chapter 2 | Loops, Lists and More Fundamentals of The Python Coding Book.

I'll also use concepts from object-oriented programming in this article. If you need to read more about this topic, you can read Chapter 7 | Object-Oriented Programming in The Python Coding Book or A Magical Tour Through Object-Oriented Programming in Python • Hogwarts School of Codecraft and Algorithmancy, the series of articles about the topic on The Python Coding Stack.

Iterating Through Iterables

Let's jump straight to what you can loop through using a for loop. The data type at the end of the for statement must be iterable. We'll see what this means in a second. However, the cheap and cheerful definition of an iterable is "anything you can use in a for loop".

Let's define a class and add to it bit by bit to make it iterable. I'll explore different routes to create a custom class you can use in a for loop.

I'll create a class called RandomWalk which can hold several values. The twist is that each time you loop through a RandomWalk object, the loop will go through the items in a random order. However, the order in which the items are stored in the object doesn't change.

Let's start by defining the class and its __init__() special method:

class RandomWalk: def __init__(self, iterable): self.values = list(iterable) captains = RandomWalk( [ "Pike", "Kirk", "Picard", "Sisko", "Janeway", "Riker", ] ) for captain in captains: print(captain) — copy code

You pass an iterable when you initialise a RandomWalk object and store it as a list in the data attribute .values. You also create an instance of this class with the names of several captains of starships from the Star Trek universe.

You use a list of strings as an argument for RandomWalk(). Does this mean you can use a RandomWalk instance in a for loop? You can run this code to find out:

 Traceback (most recent call last):
   File ... line 17, in <module>
     for captain in captains:
 TypeError: 'RandomWalk' object is not iterable

No, you cannot. The object captains is of type RandomWalk, which is not iterable. You can loop through captains.values, which is a list. But this is not the direction we'll take in this article. We'll make RandomWalk itself an iterable object.

A class needs to have the __iter__() special method defined to be iterable. Let's try adding one:

class RandomWalk: def __init__(self, iterable): self.values = list(iterable) def __iter__(self): # There's nothing in this method just yet ... captains = RandomWalk( [ "Pike", "Kirk", "Picard", "Sisko", "Janeway", "Riker", ] ) for captain in captains: print(captain) — copy code

This code still raises an error, but it's a different error this time:

 Traceback (most recent call last):
   File ... line 21, in <module>
     for captain in captains:
 TypeError: iter() returned non-iterator of type 'NoneType'

This is progress. You're no longer told that the 'RandomWalk object is not iterable' as in the previous error message since __iter__() is defined now. However, the special method __iter__() must return an iterator. The method implicitly returns None, and that's why the error message refers to a NoneType object.

In the next section, we'll distinguish between iterable and iterator properly and see how they're connected through the __iter__() special method. For good measure, we'll throw in the built-in function iter(), too.

But for now, let's cheat a bit. First, let's try to return the list self.values in the __iter__() special method:

class RandomWalk: # ... def __iter__(self): # We'll change this later return self.values # ... — copy code

This raises a similar error message as before. The difference is that message refers to the list object now:

 Traceback (most recent call last):
   File ... line 21, in <module>
     for captain in captains:
 TypeError: iter() returned non-iterator of type 'list'

A list is iterable. But it's not an iterator. Let's preview the next section by stating that we can convert an iterable to an iterator using the built-in function iter(). Note the change in the return statement:

class RandomWalk: def __init__(self, iterable): self.values = list(iterable) def __iter__(self): # We'll change this later return iter(self.values) captains = RandomWalk( [ "Pike", "Kirk", "Picard", "Sisko", "Janeway", "Riker", ] ) for captain in captains: print(captain) — copy code

This works. The code prints out the list of captains:

 Pike
 Kirk
 Picard
 Sisko
 Janeway
 Riker

The RandomWalk class is now iterable, and you can use its instances, such as captains, in a for loop. But this class just mimics a list for now. In the next section, we'll explore iterators further and create a RandomWalkIterator class.

From Iterables to Iterators

An iterable is a data structure that can be used in a for loop. More generally, it's a data structure that can return its members one at a time. There are other processes in Python, besides the for loop, which rely on this feature of iterables. These include list comprehensions and unpacking, among others.

You can always create an iterator from an iterable. One way of doing this is by passing an iterable to the built-in function iter(), which calls the object's __iter__() special method.

An iterator is an object that doesn't contain its own data. Instead, it refers to data stored in another structure. Specifically, it keeps track of which item comes next. You can think of an iterator as a disposable data structure. It goes through each item in a structure, one at a time, returning the next value each time it's needed.

Each time you iterate through an iterable, such as in a for loop, a new iterator is created from the iterable. So, if you iterate through the same iterable several times, a different iterator is used for each one.

Let's create a RandomWalkIterator class which will act as the iterator for the RandomWalk iterable. You want the order of iteration to be random each time you iterate through the RandomWalk iterable:

import random class RandomWalk: def __init__(self, iterable): self.values = list(iterable) def __iter__(self): return RandomWalkIterator(self) class RandomWalkIterator: def __init__(self, random_walk): self.random_walk = random_walk self.random_indices = list( range(len(random_walk.values)) ) random.shuffle(self.random_indices) self.idx = 0 — copy code

Let's look at the change in RandomWalk.__iter__() first. This method now returns an object of type RandomWalkIterator. You pass the RandomWalk instance itself when you create the RandomWalkIterator instance.

In RandomWalkIterator.__init__(), you create a list of integers and shuffle it. Each time you create a new RandomWalkIterator, you generate a new random order of integers which you use as indices.

You also create self.idx and set it to 0. This index will keep track of the progress through the items in the iterable. You'll use this data attribute shortly.

Does this work? You can try running this as you did earlier:

But you find an error you've encountered several times today already:

 Traceback (most recent call last):
   File ... line 32, in <module>
     for captain in captains:
 TypeError: iter() returned non-iterator of type 'RandomWalkIterator'

RandomWalk.__iter__() returns an instance of RandomWalkIterator. But just having 'Iterator' in its name is not enough to make this object an iterator!

Let's see what comes next…

What's next?

A class must have the __next__() special method to make it an iterator. This method returns the next value or raises a StopIteration exception if no more elements are left.

Let's add the __next__() special method to make RandomWalkIterator an iterator:

import random class RandomWalk: # ... class RandomWalkIterator: # ... def __next__(self): output = self.random_walk.values[ self.random_indices[self.idx] ] self.idx += 1 return output # ... — copy code

The __next__() method fetches the next index from self.random_indices and uses it to get the required value from the iterable.

Before the method returns the value, it increments the value of self.idx, ready for the next time __next__() is needed.

This almost works. Let's look at the output from this code and see whether it prints out the Star Trek captains in random order:

 Pike
 Janeway
 Picard
 Kirk
 Riker
 Sisko
 Traceback (most recent call last):
   File ... line 39, in <module>
     for captain in captains:
   File ... line 22, in __next__
     self.random_indices[self.idx]
 IndexError: list index out of range

The good news first: the code printed out the captains' names. And you'll see that each time you run the code, the order in which the names are printed is random.

However, there's also an error that shows up after the names are printed. The traceback shows that the error is in the for loop. But you can trace it back to the __next__() method, as the second message in the traceback shows.

I'll ask you to be patient for a bit longer. We'll fix this soon. But first, it's time to look deeper within the for loop.

Looking Inside the `for` Loop

Let me start with a brief description of what a for loop does:

The for loop creates an iterator from the iterable using the built-in iter() function.
It calls the iterator's __next__() method and assigns the return value to the variable in the for loop statement.
The program executes the code within the loop. Then, the loop repeats itself and calls the iterator's __next__() method again. This process keeps repeating.
If the iterator's __next__() method raises a StopIteration, the for loop terminates.

I won't test you on what you read just yet! We'll go through these steps in detail soon.

We'll get back to RandomWalk, RandomWalkIterator, and the starship captains later. For this section, I'll use a more basic demonstration example:

class SomeIterable: def __iter__(self): print("SomeIterable.__iter__() has just been called\n") return SomeIterator() class SomeIterator: def __next__(self): print("SomeIterator.__next__() has just been called") response = input("Stop iterating? [y/n] ") if response.lower() == "y": raise StopIteration my_iterable = SomeIterable() print("Ready to start the 'for' loop…\n") for item in my_iterable: pass — copy code

SomeIterable is an iterable class since it has an __iter__() method. The __iter__() method returns an instance of SomeIterator, which is an iterator because it has the __next__() method. You'll see later that an iterator should also have an __iter__() method, but this is not relevant to this part of the discussion.

Both methods leave 'breadcrumbs' behind them by printing a message whenever they're called. The iterator's __next__() method asks the user whether they want to continue iterating. Note that this is unusual for an iterator. I'm only using this example for demonstration purposes. This call to input() in SomeIterator.__next__() makes the iteration interactive to enable you to explore further.

If you type "y", the method raises a StopIteration.

In the rest of this script, you create an instance of SomeIterable and use it in a for loop. But you should note there's no code within the for loop except for a pass statement. Therefore, anything you observe when you run this code happens "behind the scenes". Let's run this code:

 Ready to start the 'for' loop…
 
 SomeIterable.__iter__() has just been called
 
 SomeIterator.__next__() has just been called
 Stop iterating? [y/n] n
 SomeIterator.__next__() has just been called
 Stop iterating? [y/n] n
 SomeIterator.__next__() has just been called
 Stop iterating? [y/n] n
 SomeIterator.__next__() has just been called
 Stop iterating? [y/n] y

The first 'breadcrumb' you see once the for loop starts is the one left by the iterable's __iter__() method. The loop creates an iterator from the iterable. This is the first step shown in the bullet points at the start of this section.

The loop then calls the iterator's __next__() method. The value returned by SomeIterator.__next__() is assigned to the variable item that's defined in the for loop statement. In this example, you're not using this variable item within the for loop.

The rest of the loop is executed. However, there are no further steps to execute in this case since pass is the only code within the loop. The process of calling the iterator's __next__() method, assigning the value it returns to item, and executing the rest of the loop is repeated for each iteration of the loop.

However, in the last iteration, you type "y", and the __next__() method raises a StopIteration exception. You don't see the error when you run the script since this exception is the signal needed by the for loop to stop iterating.

Back to `RandomWalk` and `RandomWalkIterator`

Let's recall where we left our RandomWalk example. Here's the code so far:

And the output is the following:

 Kirk
 Sisko
 Pike
 Janeway
 Riker
 Picard
 Traceback (most recent call last):
   File ... line 39, in <module>
     for captain in captains:
   File ... line 22, in __next__
     self.random_indices[self.idx]
 IndexError: list index out of range

Let's bring the knowledge from the last section into this code. The for loop will keep calling the iterator's __next__() method until there's a StopIteration exception. But the current version of RandomWalkIterator.__next__() never raises this exception. Let's fix this now:

import random class RandomWalk: # ... class RandomWalkIterator: # ... def __next__(self): if self.idx >= len(self.random_indices): raise StopIteration output = self.random_walk.values[ self.random_indices[self.idx] ] self.idx += 1 return output # ... — copy code

The __next__() method now checks whether the index is still within the required range and raises a StopIteration exception if it isn't.

Recall that each iterator is only used once. So, when you create an instance of RandomWalkIterator, the data attribute .idx starts with the value 0. This value is incremented each time the iterator's __next__() method is called.

Iterators Are Also Iterables

There's one last addition needed to tie loose ends. We've seen that an iterable is an object you can iterate through. It has an __iter__() method, which returns an iterator. An iterator has a __next__() method that returns the next item from the iterable.

It is possible for a data structure that holds data to be both an iterable and an iterator. A class is both an iterable and an iterator if it has both an __iter__() method and a __next__() method. Recall that __iter__() must return an iterator. Therefore, in such cases, __iter__() returns self, the instance itself. You'll see this in action soon.

By convention, every iterator also has an __iter__() method to make it iterable. Therefore, you can use an iterator directly in a for loop since it's also an iterable.

All iterators are iterables. But not all iterables are iterators.

Here's the final version of the code in this article. In this version, RandomWalkIterator also has an __iter__() method to make it an iterable in addition to being an iterator:

Note how RandomWalkIterator.__iter__() returns self. If you use an instance of this class as an iterable, it returns the instance itself since it's also an iterator.

A Bit of History

Way back when the dinosaurs roamed the Earth (well, not quite so far back in time, but who's counting), for loops followed a different protocol. An object was an iterable if it had the __getitem__() special method defined. You can read about this special method in another post I published recently: The Manor House, the Oak-Panelled Library, the Vending Machine, and Python's __getitem__() [Part 1].

Let's briefly look at the "behind the scenes" of a for loop using this older protocol. I'll use another example designed for demonstration purposes only:

class AnotherIterable: def __init__(self, iterable): self.values = iterable def __getitem__(self, item): print( f"AnotherIterable.__getitem__() has just " f"been called with argument {item}" ) return self.values[item] my_iterable = AnotherIterable([2, 4, 6, 8]) print("Ready to start the 'for' loop…\n") for item in my_iterable: pass — copy code

AnotherIterable has a single data attribute called .values. You pass an iterable when you create an instance of this new class. The class has the __getitem__() special method. The method prints out a statement each time it is called and returns a value from the list in self.values.

You create an instance of this class from a list with four integers and use it in a for loop. As you did in an earlier section, you don't include any code within the for loop except a pass statement. Here's the output from this code:

 Ready to start the 'for' loop…
 
 AnotherIterable.__getitem__() has just been called with argument 0
 AnotherIterable.__getitem__() has just been called with argument 1
 AnotherIterable.__getitem__() has just been called with argument 2
 AnotherIterable.__getitem__() has just been called with argument 3
 AnotherIterable.__getitem__() has just been called with argument 4

The for loop couldn't create an iterator in this case since the class doesn't have an __iter__() method. However, it falls back to using __getitem__() if this method is present. The lines printed show that __getitem__() is called five times in this case. But there are only four numbers in the list used to create this instance. The final time __getitem__() is called, it raises an exception since self.values has run out of elements. This exception, which you do not see, is used by the for loop as a signal to stop looping.

Note how this class doesn't have __iter__() defined, but you don't get an error saying that his object is not iterable when you use it in a for loop. The __getitem__() method provides a fallback protocol for iteration. This was the standard protocol before the iterator protocol was introduced in Python.

However, the iterator protocol using __iter__() to create an iterator is the standard one and has been for a long time. If a class has both __iter__() and __getitem__(), the newer iterator protocol is used. You can confirm this by adding __iter__() to AnotherIterable:

class AnotherIterable: def __init__(self, iterable): self.values = iterable def __iter__(self): return iter(self.values) def __getitem__(self, item): print( f"AnotherIterable.__getitem__() has just " f"been called with argument {item}" ) return self.values[item] my_iterable = AnotherIterable([2, 4, 6, 8]) print("Ready to start the 'for' loop…\n") for item in my_iterable: pass — copy code

The output of this code no longer shows the lines printed by __getitem__() since the method is no longer called in the for loop now that __iter__() is defined:

 Ready to start the 'for' loop…

You should always use __iter__() to make custom classes iterable since this is the newer and preferred option.

Final Words

The for loop is one of the most fundamental tools in Python programming. Most programmers use it effectively without ever needing to know how it really works. But understanding the inner workings of the tools we use gives us more flexibility and confidence when using them. Next time you use a for loop, you'll be able to visualise its inner workings better.

Code in this article uses Python 3.11

You can find a video discussing this topic here: Do You Really Understand How Python Iteration Works? The Anatomy of a `for` Loop – The Python Coding Place

Stop Stack

#16

Recently published articles on The Python Coding Stack:
- "You Have Your Mother's Eyes" • Inheritance in Python Classes. Year 5 at Hogwarts School of Codecraft and Algorithmancy • Inheritance
- Casting A Spell • More Interaction Between Classes. Year 4 at Hogwarts School of Codecraft and Algorithmancy • More on Methods and Classes
- An Object That Contains Objects • Python's Containers. Containers • Part 4 of the Data Structure Categories Series
- Zen and The Art of Python turtle Animations • A Step-by-Step Guide. Python's turtle is not just for drawing simple shapes
- The One About The Taxi Driver, Mappings, and Sequences • A Short Trip to 42 Python Street. How can London cabbies help us learn about mappings and sequences in Python?
Recently published articles on Breaking the Rules, my other substack about narrative technical writing:
- Are You Ready to Break the Rules? Narrative Technical Writing: Using storytelling techniques in technical articles
- The Different Flavours of Narrative Technical Writing. Why I'm using more storytelling techniques in my Python articles
Stats on the Stack
- Age: 2 months, 1 week, and 2 days old
- Number of articles: 16
- Subscribers: 572
There's a new page on The Python Coding Stack's homepage called The Python Series—that's series as a plural noun! It's a shame series is a word that has the same singular and plural form. For the avid readers, you know that I publish several stand-alone articles, but I also publish articles that are part of series (pl.). I'll use this page to keep all series in one place. At the moment, there are links to The Data Structure Category Series and The Harry Potter Object-Oriented Programming Series. I'll add more links to future series here, too.
Most articles will be published in full on the free subscription. However, a lot of effort and time goes into crafting and preparing these articles. If you enjoy the content and find it useful, and if you're in a position to do so, you can become a paid subscriber. In addition to supporting this work, you'll get access to the full archive of articles and some paid-only articles.