The Anatomy of a for Loop
What happens behind the scenes when you run a Python `for` loop? How complex can it be?
Learning to use the for
loop is one of the first steps in everyone's Python journey. Early on, you learn how to loop using range()
and how to loop through a list. A bit later, you discover you can loop through strings, tuples, and other iterable data types. You even learn the trick of looping through a dictionary using its items()
method.
But what's really happening behind the scenes in a for
loop?
Let's look inside the for
loop and make sense of the "iter-soup" of terms: iterate, iterable, iterator, iter()
, and __iter__()
.
I will not describe and explain how to use the for
loop in this post. If you're looking for a "for
loop primer", you can read Chapter 2 | Loops, Lists and More Fundamentals of The Python Coding Book.
I'll also use concepts from object-oriented programming in this article. If you need to read more about this topic, you can read Chapter 7 | Object-Oriented Programming in The Python Coding Book or A Magical Tour Through Object-Oriented Programming in Python • Hogwarts School of Codecraft and Algorithmancy, the series of articles about the topic on The Python Coding Stack.
Iterating Through Iterables
Let's jump straight to what you can loop through using a for
loop. The data type at the end of the for
statement must be iterable. We'll see what this means in a second. However, the cheap and cheerful definition of an iterable is "anything you can use in a for
loop".
Let's define a class and add to it bit by bit to make it iterable. I'll explore different routes to create a custom class you can use in a for
loop.
I'll create a class called RandomWalk
which can hold several values. The twist is that each time you loop through a RandomWalk
object, the loop will go through the items in a random order. However, the order in which the items are stored in the object doesn't change.
Let's start by defining the class and its __init__()
special method:
You pass an iterable when you initialise a RandomWalk
object and store it as a list in the data attribute .values
. You also create an instance of this class with the names of several captains of starships from the Star Trek universe.
You use a list of strings as an argument for RandomWalk()
. Does this mean you can use a RandomWalk
instance in a for
loop? You can run this code to find out:
 Traceback (most recent call last):
 File ... line 17, in <module>
  for captain in captains:
 TypeError: 'RandomWalk' object is not iterable
No, you cannot. The object captains
is of type RandomWalk
, which is not iterable. You can loop through captains.values
, which is a list. But this is not the direction we'll take in this article. We'll make RandomWalk
itself an iterable object.
A class needs to have the __iter__()
special method defined to be iterable. Let's try adding one:
This code still raises an error, but it's a different error this time:
 Traceback (most recent call last):
 File ... line 21, in <module>
  for captain in captains:
 TypeError: iter() returned non-iterator of type 'NoneType'
This is progress. You're no longer told that the 'RandomWalk object is not iterable'
as in the previous error message since __iter__()
is defined now. However, the special method __iter__()
must return an iterator. The method implicitly returns None
, and that's why the error message refers to a NoneType
object.
In the next section, we'll distinguish between iterable and iterator properly and see how they're connected through the __iter__()
special method. For good measure, we'll throw in the built-in function iter()
, too.
But for now, let's cheat a bit. First, let's try to return the list self.values
in the __iter__()
special method:
This raises a similar error message as before. The difference is that message refers to the list
object now:
 Traceback (most recent call last):
 File ... line 21, in <module>
  for captain in captains:
 TypeError: iter() returned non-iterator of type 'list'
A list is iterable. But it's not an iterator. Let's preview the next section by stating that we can convert an iterable to an iterator using the built-in function iter()
. Note the change in the return
statement:
This works. The code prints out the list of captains:
 Pike
 Kirk
 Picard
 Sisko
 Janeway
 Riker
The RandomWalk
class is now iterable, and you can use its instances, such as captains
, in a for
loop. But this class just mimics a list for now. In the next section, we'll explore iterators further and create a RandomWalkIterator
class.
From Iterables to Iterators
An iterable is a data structure that can be used in a for
loop. More generally, it's a data structure that can return its members one at a time. There are other processes in Python, besides the for
loop, which rely on this feature of iterables. These include list comprehensions and unpacking, among others.
You can always create an iterator from an iterable. One way of doing this is by passing an iterable to the built-in function iter()
, which calls the object's __iter__()
special method.
An iterator is an object that doesn't contain its own data. Instead, it refers to data stored in another structure. Specifically, it keeps track of which item comes next. You can think of an iterator as a disposable data structure. It goes through each item in a structure, one at a time, returning the next value each time it's needed.
Each time you iterate through an iterable, such as in a for
loop, a new iterator is created from the iterable. So, if you iterate through the same iterable several times, a different iterator is used for each one.
Let's create a RandomWalkIterator
class which will act as the iterator for the RandomWalk
iterable. You want the order of iteration to be random each time you iterate through the RandomWalk
iterable:
Let's look at the change in RandomWalk.__iter__()
first. This method now returns an object of type RandomWalkIterator
. You pass the RandomWalk
instance itself when you create the RandomWalkIterator
instance.
In RandomWalkIterator.__init__()
, you create a list of integers and shuffle it. Each time you create a new RandomWalkIterator
, you generate a new random order of integers which you use as indices.
You also create self.idx
and set it to 0
. This index will keep track of the progress through the items in the iterable. You'll use this data attribute shortly.
Does this work? You can try running this as you did earlier:
But you find an error you've encountered several times today already:
 Traceback (most recent call last):
 File ... line 32, in <module>
  for captain in captains:
 TypeError: iter() returned non-iterator of type 'RandomWalkIterator'
RandomWalk.__iter__()
returns an instance of RandomWalkIterator
. But just having 'Iterator' in its name is not enough to make this object an iterator!
Let's see what comes next…
What's next?
A class must have the __next__()
special method to make it an iterator. This method returns the next value or raises a StopIteration
exception if no more elements are left.
Let's add the __next__()
special method to make RandomWalkIterator
an iterator:
The __next__()
method fetches the next index from self.random_indices
and uses it to get the required value from the iterable.
Before the method returns the value, it increments the value of self.idx
, ready for the next time __next__()
is needed.
This almost works. Let's look at the output from this code and see whether it prints out the Star Trek captains in random order:
 Pike
 Janeway
 Picard
 Kirk
 Riker
 Sisko
 Traceback (most recent call last):
 File ... line 39, in <module>
  for captain in captains:
 File ... line 22, in __next__
  self.random_indices[self.idx]
 IndexError: list index out of range
The good news first: the code printed out the captains' names. And you'll see that each time you run the code, the order in which the names are printed is random.
However, there's also an error that shows up after the names are printed. The traceback shows that the error is in the for
loop. But you can trace it back to the __next__()
method, as the second message in the traceback shows.
I'll ask you to be patient for a bit longer. We'll fix this soon. But first, it's time to look deeper within the for
loop.
Looking Inside the for
Loop
Let me start with a brief description of what a for
loop does:
The
for
loop creates an iterator from the iterable using the built-initer()
function.It calls the iterator's
__next__()
method and assigns the return value to the variable in thefor
loop statement.The program executes the code within the loop. Then, the loop repeats itself and calls the iterator's
__next__()
method again. This process keeps repeating.If the iterator's
__next__()
method raises aStopIteration
, thefor
loop terminates.
I won't test you on what you read just yet! We'll go through these steps in detail soon.
We'll get back to RandomWalk
, RandomWalkIterator
, and the starship captains later. For this section, I'll use a more basic demonstration example:
SomeIterable
is an iterable class since it has an __iter__()
method. The __iter__()
method returns an instance of SomeIterator
, which is an iterator because it has the __next__()
method. You'll see later that an iterator should also have an __iter__()
method, but this is not relevant to this part of the discussion.
Both methods leave 'breadcrumbs' behind them by printing a message whenever they're called. The iterator's __next__()
method asks the user whether they want to continue iterating. Note that this is unusual for an iterator. I'm only using this example for demonstration purposes. This call to input()
in SomeIterator.__next__()
makes the iteration interactive to enable you to explore further.
If you type "y"
, the method raises a StopIteration
.
In the rest of this script, you create an instance of SomeIterable
and use it in a for
loop. But you should note there's no code within the for
loop except for a pass
statement. Therefore, anything you observe when you run this code happens "behind the scenes". Let's run this code:
 Ready to start the 'for' loop…
 ​
 SomeIterable.__iter__() has just been called
 ​
 SomeIterator.__next__() has just been called
 Stop iterating? [y/n] n
 SomeIterator.__next__() has just been called
 Stop iterating? [y/n] n
 SomeIterator.__next__() has just been called
 Stop iterating? [y/n] n
 SomeIterator.__next__() has just been called
 Stop iterating? [y/n] y
The first 'breadcrumb' you see once the for
loop starts is the one left by the iterable's __iter__()
method. The loop creates an iterator from the iterable. This is the first step shown in the bullet points at the start of this section.
The loop then calls the iterator's __next__()
method. The value returned by SomeIterator.__next__()
is assigned to the variable item
that's defined in the for
loop statement. In this example, you're not using this variable item
within the for
loop.
The rest of the loop is executed. However, there are no further steps to execute in this case since pass
is the only code within the loop. The process of calling the iterator's __next__()
method, assigning the value it returns to item
, and executing the rest of the loop is repeated for each iteration of the loop.
However, in the last iteration, you type "y"
, and the __next__()
method raises a StopIteration
exception. You don't see the error when you run the script since this exception is the signal needed by the for
loop to stop iterating.
Back to RandomWalk
and RandomWalkIterator
Let's recall where we left our RandomWalk
example. Here's the code so far:
And the output is the following:
 Kirk
 Sisko
 Pike
 Janeway
 Riker
 Picard
 Traceback (most recent call last):
 File ... line 39, in <module>
  for captain in captains:
 File ... line 22, in __next__
  self.random_indices[self.idx]
 IndexError: list index out of range
Let's bring the knowledge from the last section into this code. The for
loop will keep calling the iterator's __next__()
method until there's a StopIteration
exception. But the current version of RandomWalkIterator.__next__()
never raises this exception. Let's fix this now:
The __next__()
method now checks whether the index is still within the required range and raises a StopIteration
exception if it isn't.
Recall that each iterator is only used once. So, when you create an instance of RandomWalkIterator
, the data attribute .idx
starts with the value 0
. This value is incremented each time the iterator's __next__()
method is called.
Iterators Are Also Iterables
There's one last addition needed to tie loose ends. We've seen that an iterable is an object you can iterate through. It has an __iter__()
method, which returns an iterator. An iterator has a __next__()
method that returns the next item from the iterable.
It is possible for a data structure that holds data to be both an iterable and an iterator. A class is both an iterable and an iterator if it has both an __iter__()
method and a __next__()
method. Recall that __iter__()
must return an iterator. Therefore, in such cases, __iter__()
returns self
, the instance itself. You'll see this in action soon.
By convention, every iterator also has an __iter__()
method to make it iterable. Therefore, you can use an iterator directly in a for
loop since it's also an iterable.
All iterators are iterables. But not all iterables are iterators.
Here's the final version of the code in this article. In this version, RandomWalkIterator
also has an __iter__()
method to make it an iterable in addition to being an iterator:
Note how RandomWalkIterator.__iter__()
returns self
. If you use an instance of this class as an iterable, it returns the instance itself since it's also an iterator.
A Bit of History
Way back when the dinosaurs roamed the Earth (well, not quite so far back in time, but who's counting), for
loops followed a different protocol. An object was an iterable if it had the __getitem__()
special method defined. You can read about this special method in another post I published recently: The Manor House, the Oak-Panelled Library, the Vending Machine, and Python's __getitem__() [Part 1].
Let's briefly look at the "behind the scenes" of a for
loop using this older protocol. I'll use another example designed for demonstration purposes only:
AnotherIterable
has a single data attribute called .values
. You pass an iterable when you create an instance of this new class. The class has the __getitem__()
special method. The method prints out a statement each time it is called and returns a value from the list in self.values
.
You create an instance of this class from a list with four integers and use it in a for
loop. As you did in an earlier section, you don't include any code within the for
loop except a pass
statement. Here's the output from this code:
 Ready to start the 'for' loop…
 ​
 AnotherIterable.__getitem__() has just been called with argument 0
 AnotherIterable.__getitem__() has just been called with argument 1
 AnotherIterable.__getitem__() has just been called with argument 2
 AnotherIterable.__getitem__() has just been called with argument 3
 AnotherIterable.__getitem__() has just been called with argument 4
The for
loop couldn't create an iterator in this case since the class doesn't have an __iter__()
method. However, it falls back to using __getitem__()
if this method is present. The lines printed show that __getitem__()
is called five times in this case. But there are only four numbers in the list used to create this instance. The final time __getitem__()
is called, it raises an exception since self.values
has run out of elements. This exception, which you do not see, is used by the for
loop as a signal to stop looping.
Note how this class doesn't have __iter__()
defined, but you don't get an error saying that his object is not iterable when you use it in a for
loop. The __getitem__()
method provides a fallback protocol for iteration. This was the standard protocol before the iterator protocol was introduced in Python.
However, the iterator protocol using __iter__()
to create an iterator is the standard one and has been for a long time. If a class has both __iter__()
and __getitem__()
, the newer iterator protocol is used. You can confirm this by adding __iter__()
to AnotherIterable
:
The output of this code no longer shows the lines printed by __getitem__()
since the method is no longer called in the for
loop now that __iter__()
is defined:
 Ready to start the 'for' loop…
 ​
You should always use __iter__()
to make custom classes iterable since this is the newer and preferred option.
Final Words
The for
loop is one of the most fundamental tools in Python programming. Most programmers use it effectively without ever needing to know how it really works. But understanding the inner workings of the tools we use gives us more flexibility and confidence when using them. Next time you use a for
loop, you'll be able to visualise its inner workings better.
Code in this article uses Python 3.11
You can find a video discussing this topic here: Do You Really Understand How Python Iteration Works? The Anatomy of a `for` Loop – The Python Coding Place
Stop Stack
#16
Recently published articles on The Python Coding Stack:
"You Have Your Mother's Eyes" • Inheritance in Python Classes. Year 5 at Hogwarts School of Codecraft and Algorithmancy • Inheritance
Casting A Spell • More Interaction Between Classes. Year 4 at Hogwarts School of Codecraft and Algorithmancy • More on Methods and Classes
An Object That Contains Objects • Python's Containers. Containers • Part 4 of the Data Structure Categories Series
Zen and The Art of Python
turtle
Animations • A Step-by-Step Guide. Python'sturtle
is not just for drawing simple shapesThe One About The Taxi Driver, Mappings, and Sequences • A Short Trip to 42 Python Street. How can London cabbies help us learn about mappings and sequences in Python?
Recently published articles on Breaking the Rules, my other substack about narrative technical writing:
Are You Ready to Break the Rules? Narrative Technical Writing: Using storytelling techniques in technical articles
The Different Flavours of Narrative Technical Writing. Why I'm using more storytelling techniques in my Python articles
Stats on the Stack
Age: 2 months, 1 week, and 2 days old
Number of articles: 16
Subscribers: 572
There's a new page on The Python Coding Stack's homepage called The Python Series—that's series as a plural noun! It's a shame series is a word that has the same singular and plural form. For the avid readers, you know that I publish several stand-alone articles, but I also publish articles that are part of series (pl.). I'll use this page to keep all series in one place. At the moment, there are links to The Data Structure Category Series and The Harry Potter Object-Oriented Programming Series. I'll add more links to future series here, too.
Most articles will be published in full on the free subscription. However, a lot of effort and time goes into crafting and preparing these articles. If you enjoy the content and find it useful, and if you're in a position to do so, you can become a paid subscriber. In addition to supporting this work, you'll get access to the full archive of articles and some paid-only articles.