Great post. Most python devs don't realize that pretty much everything in cpython uses C pointers and it's useful to understand how they work. I wrote this reply to a question on variable swapping the other day and think it is a useful complement to your post:
Variable swapping:
a,b = b,a
The commenter asked if temp variables are used in the bytecode/c code to make this bit of magic happen. Here was my response:
In the C language python implementation (cpython) tuples (righthand b,a above) use pointers to PyObjects. In order to achieve the swapping behavior the C code creates a new tuple which consist of the same PyObjects but with new pointers (call them a' and b') and these are used to assign b' to a and a' to b. So in effect two 'temp vars' are created but they are lightweight pointers thus very cheap.
On a related note it's important to remember that in cpython parameters are passed as "reference by value". This means that any var x passed into a method and reassigned a new value won't be reflected in the variable outside that call stack.
x=5
def modX(x: int):
x = 10
return x
y=modX(x)
assert x == 5
assert y==10
Of course if there is no reassignment inside the method this doesn't happen. If instead you pass a class object and update some of its values, e.g., a dict key addition, then the reference outside the function will still point to that updated dict - since as Stephen points out a dict and most class instances are mutable.
Similar to the variable swapping example, the reference (a C pointer for our purposes) is passed "by value" which means it's a copy of a pointer (holding a memory address), not the address space that holds the pointer itself. Thus when a new address is assigned to the reference/pointer inside the function it is not reflected in the pointer variable outside the call stack.
In order to have that effect we would need to perform the call thus:
x = modX(x)
since then the pointer to x outside the call stack will be reassigned to the copied pointer passed into modX.
Great post. Most python devs don't realize that pretty much everything in cpython uses C pointers and it's useful to understand how they work. I wrote this reply to a question on variable swapping the other day and think it is a useful complement to your post:
Variable swapping:
a,b = b,a
The commenter asked if temp variables are used in the bytecode/c code to make this bit of magic happen. Here was my response:
In the C language python implementation (cpython) tuples (righthand b,a above) use pointers to PyObjects. In order to achieve the swapping behavior the C code creates a new tuple which consist of the same PyObjects but with new pointers (call them a' and b') and these are used to assign b' to a and a' to b. So in effect two 'temp vars' are created but they are lightweight pointers thus very cheap.
On a related note it's important to remember that in cpython parameters are passed as "reference by value". This means that any var x passed into a method and reassigned a new value won't be reflected in the variable outside that call stack.
x=5
def modX(x: int):
x = 10
return x
y=modX(x)
assert x == 5
assert y==10
Of course if there is no reassignment inside the method this doesn't happen. If instead you pass a class object and update some of its values, e.g., a dict key addition, then the reference outside the function will still point to that updated dict - since as Stephen points out a dict and most class instances are mutable.
Similar to the variable swapping example, the reference (a C pointer for our purposes) is passed "by value" which means it's a copy of a pointer (holding a memory address), not the address space that holds the pointer itself. Thus when a new address is assigned to the reference/pointer inside the function it is not reflected in the pointer variable outside the call stack.
In order to have that effect we would need to perform the call thus:
x = modX(x)
since then the pointer to x outside the call stack will be reassigned to the copied pointer passed into modX.
It's always fun to dive into what's happening behind the scenes! Thanks for adding more to this discussion.
The way data is passed into functions is, as you say, another topic that can lead to confusion. I tried writing a bit about this a few weeks ago, too, in https://www.thepythoncodingstack.com/p/python-pass-by-value-reference-assignment