The Magician's Sleight of Hand

A Python switcheroo that will leave the audience stunned

Pulling rabbits out of hats is getting a bit boring. Pygini the Magnificent (that's his stage name–his real name is Joe) wants to update his act.

He asked me to try out his latest act on my audience here at The Python Coding Stack. So, here it is.

462,709 Lines of Code

We start with an application that has a total of 462,709 lines of code. Here's the code:

All code blocks are available in text format at the end of this article • #1

Joe insisted I show you all 462,709 lines of code, but I took the executive decision not to do this. You can thank me later. These are the highlights that matter for this "magic trick":

• The function create_display() is defined, and it returns a string.

• There are many calls to this function scattered across the whole code base. Only three calls are shown above.

If Joe asks, tell him you read through the whole code!

Here's the output from this code:

Chocolate • Quantity:  3 • Total: £      3.60
Laptop    • Quantity:  1 • Total: £  1,250.00
Tea       • Quantity:  2 • Total: £      1.80

Your Mission, Should You Choose to Accept It...

Here are Joe's rules. You need to change the displayed text showing the items, quantities, and total cost so that the text is all uppercase. But there are some restrictions on what you can and cannot do:

• This code is used widely by many people, so you cannot change the function definition for create_display()—Not even a small change.

• And you can't change the calls to create_display() either. There are many of these calls scattered across the code.

To put it in another way, you cannot make any changes to any of the lines shown above. You can add code, but you can't change or remove any code that already exists.

And one more thing, you should be able to easily reverse the change.

Do you accept this challenge? Or do you think it's Mission Impossible?

Note: if you know what's coming, well, then you know–good for you! This article is intended for those who don't. But you can still enjoy it even if you do know.

Take some time to think about this problem before reading on.

I’m starting a series of free live workshops, The Python Coding Workshops. See the first two sessions—and I’ll add more soon—and sign up for free through the link below.

The Magician's Switcheroo

There's a reason Joe is interested in this example. It seems like an impossible task that can only be solved using magic. However, as a magician, Joe knows that even the best magic is just a trick. The success of a magic act lies in finding a way of fooling your audience.

Let's remind ourselves what we cannot do.

• We cannot change the string returned by create_display() since we cannot make any changes to the function.

• We cannot change the calls to create_display() throughout the code, either.

This is where we need some sleight of hand–the magician's switcheroo. We need to change the function without anyone noticing!

Functions Are Objects. They Have Names

Everything is an object in Python—I'm sure you've heard this phrase before! Therefore, functions are objects, too. And there's a distinction between the object and its name. Any name can be reassigned to something else if you wish, and you can also do this with the names of functions. Here's an example you should never copy:

#2

In this example, you reused the name print and reassigned it to the integer 42. Therefore, the name print no longer refers to the much-loved function formerly-known-as-print!

Can we use the same trick with the 462,709-line code base and the create_display() function?

Yes, we can. We can redefine the name create_display so it refers to a different object.

But what should we replace it with? Here are the requirements for what the object that the name create_display refers to should be and do:

1. create_display should be a function (or at least a callable) since this name is called elsewhere in the code. You need to be able to place parentheses after the name.

2. The new function needs to accept three arguments to ensure the calls in the code are valid.

3. The new function must return a string that's similar to the one returned by the "old" function. However, the string returned by the new function should be uppercase.

---

Note: In the following sections, I'll build the code in stages, some of which will include partially completed code blocks that either raise exceptions or do not produce the desired output. So don't be alarmed if you try to run bits of the code that don't work. The final versions will work fine, of course!

---

Reassigning the Name create_display

I'll start by writing a line that may not make much sense at first. Bear with me. Let's reassign the name create_display using another function, which I'll call uppercase(). Note that I'm not defining uppercase() in the example below–I'll get to this soon:

#3

I haven't changed the original function, but I reassigned the name create_display to something else. On the right-hand side of the equals sign there's a call to the as-yet-undefined uppercase() function, and create_display is passed as an argument. Note that the name create_display used within uppercase() still refers to the "old" function since it's executed before the reassignment.

Next, we'll define uppercase().

Defining uppercase()

Right, so we reused the name create_display, but we used another function called uppercase(). I'll define this function next. Since I passed another function to it when I called it, uppercase() should accept another function as an argument:

#4

The parameter name func is often used when the parameter refers to a function, so I used it here. Since functions are objects, we can pass them to other functions.

And functions can also return other functions. In this case, uppercase() needs to return a function since whatever is returned by uppercase() is assigned to create_display. And recall the requirements we discussed above: create_display needs to be a function even after the switcheroo:

#5

The function uppercase() now returns a new function, which I'm calling new_func. But I haven't created this new function yet.

As long as new_func is a function, this will satisfy the first of the three conditions listed earlier: the name create_display needs to be a function even after the magic trick!

Next, we need to define new_func, the new function that will replace the "old" create_display() function.

Define the New Function new_func

The definition of the new function new_func is only needed within the definition of uppercase() as the latter function returns the former. And since this new function new_func() must accept the same arguments as the "old" create_display() function, I can define it as follows:

#6

I'll make some changes to the function signature for new_func() later. But for now, this version makes the discussion clearer (and this is a topic that generally lacks clarity!)

What do we need new_func() to do? This new function needs to perform the same action as the "old" function create_display() but convert the final string to uppercase. Before I write the code, a reminder that within uppercase(), I'm using the parameter name func to refer to the "old" create_display() function. Why not use the same name? We'll get to this later:

#7

Let's look at the whole uppercase() function definition:

• We define uppercase() as a function that takes another function as its only argument.

• We define a new function new_func() inside uppercase(). This function definition can only be accessed from within uppercase().

• This new function new_func() has the same parameters as the "old" function create_display(). We'll change this later. But for now, this will do.

• The new function new_func() calls the "old" function, which is referred to as func within uppercase(). The arguments passed to new_func() are passed directly to func(). There are lots of functions to confuse us!

• The new function new_func() returns the uppercase version of whatever the "old" function returns.

• Finally, uppercase() returns the new function new_func. Note that it doesn't call new_func. Therefore, uppercase() doesn't return the value returned by new_func(). Instead, it returns the function itself.

You may need to reread those bullet points a few times. If you find this confusing, join the club—everyone does at first!

So, What Did We Replace the "Old" create_display() With?

Let's get back to the first line I added when we started this switcheroo magic trick:

#8

This is the magician's sleight of hand. It's why Joe loves this example and wants to use it as part of his magic act. This line calls uppercase() and passes the "old" function create_display to it. You've seen in the previous section that uppercase() returns a new function that does the same thing as the "old" function but also converts the string to uppercase. This new function is assigned to the name create_display.

So, the name create_display referred to the original function before this change. It doesn't anymore. Now, it refers to the new function returned by uppercase(). The new function still uses the code in the "old" function, but it also does a bit more.

If you run the code now, this is the output you'll see:

CHOCOLATE • QUANTITY:  3 • TOTAL: £      3.60
LAPTOP    • QUANTITY:  1 • TOTAL: £  1,250.00
TEA       • QUANTITY:  2 • TOTAL: £      1.80

Same output as earlier except for one detail: all the letters are now uppercase.

Remember the three conditions we set up earlier. Let me show them again here so you don't have to keep scrolling up and down:

1. create_display should be a function (or at least a callable) since this name is called elsewhere in the code. You need to be able to place parentheses after the name.

2. The new function needs to accept three arguments to ensure the calls in the code are valid.

3. The new function must return a string that's similar to the one returned by the "old" function. However, the string returned by the new function should be uppercase.

The name create_display is still a function since it's now the new function returned by uppercase(). Therefore, the first condition is met.

The new function accepts the same arguments as the old one because the definition of new_func() inside uppercase() ensures this—we'll improve this later, but this already meets the second condition.

And the output from this code shows that the third condition is also met: the new function returns a string similar to the "old" function, but with all the letters in uppercase.

We didn't change any of the code that was already there. All we did was define a new function called uppercase() and add the line which calls uppercase() to reassign its return value to create_display.

And this change is easily reversible: all you need to do is to comment out the line that reassigns to create_display:

#9

Without this line, the switch doesn't happen and create_display still refers to the original function.

Joe thinks this deceitful switch is better than when he puts a white scarf in his hat and then pulls out a red one a bit later, to the astonishment of his audience!

Introduction

"Hey Stephen, you've got a mistake in this article. You pasted the introduction in the wrong place. Pay attention next time, please!"

We have a term to describe this switcheroo. In the example above, we decorated the function create_display(). The decoration converts the string returned by the function to uppercase. You'll see another example of decoration later in this article. And uppercase() is the decorator function.

Why is the introduction here instead of at the top? The honest answer is that I wanted to trick you! Decorators are a complex topic, and it's fair to say they are not the easiest topic to understand. There was a long period earlier in my own journey learning Python when I would simply skip anything that said "decorator" in it. Too weird and obscure, I thought.

I didn't want you to do the same with this article. So, I haven't included the term "decorator" in the title or the subtitle or anywhere in this article until now. Sorry for deceiving you! But then, we're talking about the magician's sleight-of-hand tricks, after all, which are an accepted form of deceit!

If you've read this far, why not continue to the end of this article now?

Let's Add More Mystery to the Magic

Here's the code we have so far:

#10

The additions to the original code are the following:

• The definition of the decorator function uppercase() in the block that starts with def uppercase(func).

• The decoration of the function create_display using the decorator uppercase(). The decoration happens in the following line:

#11

However, there's a special notation you can use to decorate a function instead of using this line. Here's the code again without this assignment but with an extra line just before the definition of create_display():

#12

This code produces the same output as the previous version. The @uppercase notation is decorating the create_display() function using the uppercase() decorator.

There's Another Function Requiring the Same Treatment In The 462,709 Lines of Code

There's another function in Joe's code called create_all_items_list(), which takes an iterable with item names and creates a string showing all the items. Here it is:

#13

This segment of the code outputs the following:

The list of all items is:
Chocolate
Laptop
Tea

What if we need to decorate this function in the same way as the other one so that its output is also converted to uppercase?

Let's try it out:

#14

Unfortunately, this doesn't work. It raises the following TypeError:

Traceback (most recent call last):
  ...
TypeError: uppercase.<locals>.new_func() missing 2 
    required positional arguments: 'price' and 'qty'

We decorated create_all_items_list() with uppercase(). Let's refresh our memories about what this means:

• uppercase() is called with create_all_items_list as its argument. The parameter name func within uppercase() refers to create_all_items_list.

• Within uppercase(), a new function called new_func() is defined. This new function calls create_all_items_list() from within it. Recall that func refers to create_all_items_list. However, new_func isn't called yet. We'll get back to what happens when it's called later.

• uppercase() returns the new function new_func, which is assigned to the name create_all_items_list.

Therefore, create_all_items_list() now calls the new function new_func() instead of the "old" function create_all_items_list(). But new_func() has three parameters, whereas create_all_items_list() is called with only one argument. This explains the error message, which points out that two required positional arguments are missing.

Do we need to create a new decorator to suit create_all_items_list()? That would be wasteful since the new decorator would perform the same actions as the one we already have.

The challenge we're facing now is that we require new_func() to accept either one or three arguments depending on whether we're decorating create_all_items_list() or create_dispay(). In fact, we can make new_func() even more flexible and accept any number of arguments. This means you can use it with any function as long as the function you're decorating returns a string. This is a perfect use case for *args and **kwargs:

#15

Notice the change in two places. The parameters in the function signature for new_func() are now *args and **kwargs. But the arguments in the call to func() have also changed to *args and **kwargs. This means that any arguments passed to the original function are passed directly on to the new function new_func().

You can read more about *args and **kwargs here: Argh! What are args and kwargs in Python?

And when you run the code now, the decoration works for both functions, even though they require a different number of arguments:

THE LIST OF ALL ITEMS IS:
CHOCOLATE
LAPTOP
TEA
CHOCOLATE • QUANTITY:  3 • TOTAL: £      3.60
LAPTOP    • QUANTITY:  1 • TOTAL: £  1,250.00
TEA       • QUANTITY:  2 • TOTAL: £      1.80

The first four lines are the output from the decorated create_all_items_list() function, and the final three lines are the output from the decorated create_display() function.

And you can toggle the lines containing @uppercase with comments to turn the decoration on or off.

I named the new function new_func when defining it inside uppercase(). You'll often see this function named wrapper in decorators, but there's nothing special about the name wrapper. I'll keep my version named new_func to highlight that this is a new function that replaces the "old" one.

How Many new_func() Functions Are There?

In the current version of the code, we have decorated two functions using the same decorator. Therefore, the uppercase() function is called twice. A decorator function, like uppercase(), is called when the decorated function is defined, not each time the decorated function is called.

Therefore, there are two versions of new_func created:

• When we decorate create_display() with the uppercase() decorator, we call the uppercase() decorator function with create_display as its arguments. The decorator function call creates new_func, which calls create_display() and adds the decoration. uppercase() then returns new_func and assigns it to create_display.

• When we decorate create_all_items_list() with the same uppercase() decorator, the uppercase() function is called a second time. This time, its argument is create_all_items_list. Therefore, the new_func() function created this time is different from the previous new_func() defined when we decorated the first function. This one is assigned to create_all_items_list.

This is why this new function, new_func(), has a generic name and is defined within uppercase() and not within the global program scope. It is used to "replace" any function that's decorated using uppercase().

Counting Function Calls

Let's finish with another example. You want to keep track of how often create_display() is called in the code. One way to achieve this is with another decorator.

Let's define a new decorator called count_calls():

#16

The decorator count_calls() accepts a function as an argument. I created a local variable called counter within count_calls() and set it to zero. This variable will be initialised only once each time the decorator decorates a function. Therefore, this variable will be created at the time the decoration occurs.

If we increment this counter each time the decorated function is called, then we'll have a tally of the number of calls. Each time the decorated function is called, the function new_func(), defined in the decorator, is called. Therefore, we can increment counter inside new_func(). We can also print some text to the screen to show the number of calls. We can now try to use this decorator by decorating create_display():

#17

I removed the @uppercase decorator from create_display() for clarity, but you can leave both decorators in place, if you prefer, one on each line. Note that new_func() in count_calls() returns the output from func() directly since the decoration doesn't change the output of the function in this case.

Unfortunately, this doesn't work:

Traceback (most recent call last):
  ...
  File ..., line 12, in new_func
    counter += 1
    ^^^^^^^
UnboundLocalError: cannot access local variable
    'counter' where it is not associated with a value

Variables created inside a function definition are local to the function. You first create counter in count_calls(). But then, you also try to create another variable called counter in new_func() when you use the += operator. Recall that the expression counter += 1 is roughly similar to counter = counter + 1. This counter variable is a different variable since it's created within new_func(), and therefore, it's local to new_func(). However, the += operator needs the variable to already exist and needs access to the previous value of counter to increment its value.

We need to ensure that the counter used in new_func() is the same counter used in count_calls(). We do not need counter to be a global variable available everywhere in the program. Instead, we only need to ensure that the counter in new_func() is the same object as the counter in its enclosing function, count_calls(). The nonlocal keyword achieves this:

#18

And this now works. But before running the code, let me add back the @uppercase decorator. Here's the code in full:

#19

And here's the output:

THE LIST OF ALL ITEMS IS:
CHOCOLATE
LAPTOP
TEA
The function was called 1 times
CHOCOLATE • QUANTITY:  3 • TOTAL: £      3.60
The function was called 2 times
LAPTOP    • QUANTITY:  1 • TOTAL: £  1,250.00
The function was called 3 times
TEA       • QUANTITY:  2 • TOTAL: £      1.80

Note how there's a tally kept each time create_display() is called.

The Magic Ends Here

Joe is pleased. As a magician, he feels that decorators are a perfect addition to his magic act. Instead of switching a red scarf for a white one, he's switching one function for another without anyone noticing! The decorator performs the role of the magician, swapping the functions "out of sight".

But like all magic tricks, once you know how they're done, they lose all their magic. The same applies to decorators. There's nothing magical about them. Magicians are not allowed to reveal their tricks to the general public. But I have no such restrictions, which is why I shared this article with all of you!

There's more I could say about decorators, but I won't! I introduced the topic by stealth, so I better not push my luck!

Code in this article uses Python 3.12

---

Stop Stack

#60

• The Python Coding Book is available (Ebook and paperback). This is the First Edition, which follows from the "Zeroth" Edition that has been available online for a while—Just ask Google for "python book"!

• And if you read the book already, I'd appreciate a review on Amazon. These things matter so much for individual authors!

• If you read my articles often, and perhaps my posts on social media, too, you've heard me talk about The Python Coding Place several times. But you haven't heard me talk a lot about is Codetoday Unlimited, a platform for teenagers to learn to code in Python. The beginner levels are free so everyone can start their Python journey. If you have teenage daughters or sons, or a bit younger, too, or nephews and nieces, or neighbours' children, or any teenager you know, really, send them to Codetoday Unlimited so they can start learning Python or take their Python to the next level if they've already covered some of the basics.

• Each article is the result of years of experience and many hours of work. Hope you enjoy each one and find them useful. If you're in a position to do so, you can support this Substack further with a paid subscription. In addition to supporting this work, you'll get access to the full archive of articles. Alternatively, if you become a member of The Python Coding Place, you'll get access to all articles on The Stack as part of that membership. Of course, there's plenty more at The Place, too.

---

Appendix: Code Blocks

Code Block #1

# There are 462,709 lines of code

# Somewhere in the code:
def create_display(item, price, qty):
    return f"{item:9} • Quantity: {qty:2} • Total: £{price*qty:10,.2f}"

# Elsewhere in the code
print(create_display("Chocolate", 1.2, 3))

# Somewhere else…
print(create_display("Laptop", 1250, 1))

# And later still…
print(create_display("Tea", 0.9, 2))

# There are many more calls to 'create_display()' in the code

Code Block #2

print = 42
print("Hello!")
# Traceback (most recent call last):
#   ...
# TypeError: 'int' object is not callable
print
# 42

Code Block #3

# There are 462,709 lines of code

# Somewhere in the code:
def create_display(item, price, qty):
    return f"{item:9} • Quantity: {qty:2} • Total: £{price*qty:10,.2f}"

create_display = uppercase(create_display)

# ... Rest of code

Code Block #4

# There are 462,709 lines of code

def uppercase(func):
    ...

# Somewhere in the code:
def create_display(item, price, qty):
    return f"{item:9} • Quantity: {qty:2} • Total: £{price*qty:10,.2f}"

create_display = uppercase(create_display)

# ... Rest of code

Code Block #5

# There are 462,709 lines of code

def uppercase(func):
    ...
    return new_func

# Somewhere in the code:
def create_display(item, price, qty):
    return f"{item:9} • Quantity: {qty:2} • Total: £{price*qty:10,.2f}"

create_display = uppercase(create_display)

# ... Rest of code

Code Block #6

# There are 462,709 lines of code

def uppercase(func):
    def new_func(item, price, qty):
        ...
    return new_func

# Somewhere in the code:
def create_display(item, price, qty):
    return f"{item:9} • Quantity: {qty:2} • Total: £{price*qty:10,.2f}"

create_display = uppercase(create_display)

# ... Rest of code

Code Block #7

# There are 462,709 lines of code

def uppercase(func):
    def new_func(item, price, qty):
        output = func(item, price, qty)
        return output.upper()
    return new_func

# Somewhere in the code:
def create_display(item, price, qty):
    return f"{item:9} • Quantity: {qty:2} • Total: £{price*qty:10,.2f}"

create_display = uppercase(create_display)

# Elsewhere in the code
print(create_display("Chocolate", 1.2, 3))

# Somewhere else…
print(create_display("Laptop", 1250, 1))

# And later still…
print(create_display("Tea", 0.9, 2))

# There are many more calls to 'create_display()' in the code

Code Block #8

create_display = uppercase(create_display)

Code Block #9

# create_display = uppercase(create_display)

Code Block #10

# There are 462,709 lines of code

def uppercase(func):
    def new_func(item, price, qty):
        output = func(item, price, qty)
        return output.upper()
    return new_func

# Somewhere in the code:
def create_display(item, price, qty):
    return f"{item:9} • Quantity: {qty:2} • Total: £{price*qty:10,.2f}"

create_display = uppercase(create_display)

# Elsewhere in the code
print(create_display("Chocolate", 1.2, 3))

# Somewhere else…
print(create_display("Laptop", 1250, 1))

# And later still…
print(create_display("Tea", 0.9, 2))

# There are many more calls to 'create_display()' in the code

Code Block #11

create_display = uppercase(create_display)

Code Block #12

# There are 462,709 lines of code

def uppercase(func):
    def new_func(item, price, qty):
        output = func(item, price, qty)
        return output.upper()
    return new_func

# Somewhere in the code:
@uppercase
def create_display(item, price, qty):
    return f"{item:9} • Quantity: {qty:2} • Total: £{price*qty:10,.2f}"

# This decoration is no longer needed. It's been replaced by
# the @uppercase notation above
# create_display = uppercase(create_display)

# ... Rest of code

Code Block #13

# ...

def create_all_items_list(iterable):
    result = "The list of all items is:\n"
    return result + "\n".join(iterable)

print(create_all_items_list(["Chocolate", "Laptop", "Tea"]))

# ...

Code Block #14

# There are 462,709 lines of code

def uppercase(func):
    def new_func(item, price, qty):
        output = func(item, price, qty)
        return output.upper()
    return new_func

# ...

@uppercase
def create_all_items_list(iterable):
    result = "The list of all items is:\n"
    return result + "\n".join(iterable)

print(create_all_items_list(["Chocolate", "Laptop", "Tea"]))

# ... Rest of code

Code Block #15

# There are 462,709 lines of code

def uppercase(func):
    def new_func(*args, **kwargs):
        output = func(*args, **kwargs)
        return output.upper()
    return new_func

# Somewhere in the code:
@uppercase
def create_display(item, price, qty):
    return f"{item:9} • Quantity: {qty:2} • Total: £{price*qty:10,.2f}"

@uppercase
def create_all_items_list(iterable):
    result = "The list of all items is:\n"
    return result + "\n".join(iterable)

print(create_all_items_list(["Chocolate", "Laptop", "Tea"]))

# ... Rest of code

Code Block #16

# There are 462,709 lines of code

def uppercase(func):
    # ...

def count_calls(func):
    counter = 0
    def new_func(*args, **kwargs):
        ...

# ..

Code Block #17

# There are 462,709 lines of code

def uppercase(func):
    def new_func(*args, **kwargs):
        output = func(*args, **kwargs)
        return output.upper()
    return new_func

def count_calls(func):
    counter = 0
    def new_func(*args, **kwargs):
        counter += 1
        print(f"The function was called {counter} times")
        return func(*args, **kwargs)
    return new_func

# Somewhere in the code:
@count_calls
def create_display(item, price, qty):
    return f"{item:9} • Quantity: {qty:2} • Total: £{price*qty:10,.2f}"

# ... Rest of code

Code Block #18

# There are 462,709 lines of code

def uppercase(func):
    def new_func(*args, **kwargs):
        output = func(*args, **kwargs)
        return output.upper()
    return new_func

def count_calls(func):
    counter = 0
    def new_func(*args, **kwargs):
        nonlocal counter
        counter += 1
        print(f"The function was called {counter} times")
        return func(*args, **kwargs)
    return new_func

# Somewhere in the code:
@count_calls
def create_display(item, price, qty):
    return f"{item:9} • Quantity: {qty:2} • Total: £{price*qty:10,.2f}"

# ... Rest of code

Code Block #19

# There are 462,709 lines of code

def uppercase(func):
    def new_func(*args, **kwargs):
        output = func(*args, **kwargs)
        return output.upper()
    return new_func

def count_calls(func):
    counter = 0
    def new_func(*args, **kwargs):
        nonlocal counter
        counter += 1
        print(f"The function was called {counter} times")
        return func(*args, **kwargs)
    return new_func

# Somewhere in the code:
@count_calls
@uppercase
def create_display(item, price, qty):
    return f"{item:9} • Quantity: {qty:2} • Total: £{price*qty:10,.2f}"

@uppercase
def create_all_items_list(iterable):
    result = "The list of all items is:\n"
    return result + "\n".join(iterable)

print(create_all_items_list(["Chocolate", "Laptop", "Tea"]))

# Elsewhere in the code
print(create_display("Chocolate", 1.2, 3))

# Somewhere else…
print(create_display("Laptop", 1250, 1))

# And later still…
print(create_display("Tea", 0.9, 2))

# There are many more calls to 'create_display()' in the code