Ned Batchelder - Facts and Myths about Python names and values - PyCon 2015

By: PyCon 2015

531   3   30752

Uploaded on 04/11/2015

"Speaker: Ned Batchelder

The behavior of names and values in Python can be confusing. Like many parts of Python, it has an underlying simplicity that can be hard to discern, especially if you are used to other programming languages. Here I'll explain how it all works, and present some facts and myths along the way. Call-by-reference? Call-by-value? The answer will be clear!

Slides can be found at: https://speakerdeck.com/pycon2015 and https://github.com/PyCon/2015-slides"

Comments (12):

By anonymous    2017-09-20

Normally I would expect data to stay untouched when passing it to a function as this has its own separate namespace.

x in the function and data at the module level are two names for the same object. Since that object is mutable, any changes made to it will be "seen" regardless of which name is used to refer to the object. Namespaces can't protect you from that.

x /= 10 divides every element of the NumPy array by 10. The original data is gone after this line executes. If you were to run f(data) a few more times, you'd find the contents draw closer to 0.0 each time.

Lists are a more familiar example of the same effect:

l = list(range(4))
print(l)
# [0, 1, 2, 3]
l += [4]
print(l)
# [0, 1, 2, 3, 4]

For a good overview of this sort of thing (including related issues) I recommend Ned Batchelder's “Facts and Myths about Python Names and Values” (26 minute video from PyCon US 2015). His example of list "addition" starts about 10 minutes in.

Behind the Scenes

/ and /= (and similar pairs of operators) do different things. Tutorials often claim that these two operations are the same:

x = x / 10
x /= 10

...but they're not. Full details can be found in The Python Language Reference, 3.3.7. Emulating Numeric Types.

/ calls the __truediv__ (or maybe __rtruediv__ --- a topic for another day) method on one of the two objects, feeding the other object as the argument:

# x = x / 10
x = x.__truediv__(10)

Typically, these methods return some new value without altering the old one. This is why data was unchanged by x / 10, but id(x) changed --- x now referred to a new object, and was no longer an alias for data.

/= calls a completely different method, __itruediv__ for the "in-place" operation:

# x /= 10
x = x.__itruediv__(10)

These methods typically modify the object, which then returns self. This explains why id(x) was unchanged and why data's contents had changed --- x and data were still the one and only object. From the docs I linked above:

These methods should attempt to do the operation in-place (modifying self) and return the result (which could be, but does not have to be, self). If a specific method is not defined, the augmented assignment falls back to the normal methods [meaning __add__ and family --- KJC].

If you look at the methods of different data types, you'll find that they don't support all of these.

  • dir(0) shows that integers lack the in-place methods, which shouldn't be surprising, because they're immutable.

  • dir([]) reveals only two in-place methods: __iadd__ and __imul__ --- you can't divide or subtract from a list, but you can in-place add another list, and you can multiply it by an integer. (Again, those methods can do whatever they want with their arguments, including refuse them... list.__iadd__ won't take an integer, while list.__imul__ will reject a list.)

  • dir(np.linspace(0, 1, 5)) shows basically all of the arithmetic, logic, and bitwise methods, with normal and in-place for each. (It could be missing some --- I didn't count them all.)

Finally, to re-reiterate, what namespace these objects are in when their methods get called makes absolutely no difference. In Python, data has no scope... if you have a reference to it, you can call methods on it. (From Ned Batchelder's talk: Variables have a scope, but no type; data has a type, but no scope.)

Original Thread

By anonymous    2017-09-20

Your mental model is pretty much correct. However, you should not worry about whether Python is "pass by value" or "pass by reference". You have to learn how assignment in Python works. This is hands down the best resource to do so. The most important fact to know is that assignment never copies data.

Once you understand assignment, the only thing you have to know is that function parameters are passed by assignment.

The first thing that happens implicitly when you call your function is

NN = unsort_letters

You passed in unsort_letters as the argument. You assign another name (NN) to that argument - and that's it, no data is copied.

Is there ever a circumstance when I cannot change the contents of the passed parameter within the function

Yes, when whatever you pass in is immutable. Integers, for example, have no methods to update their value, so you can't mutate them in the function body (or anywhere else).

It is important to note however that Python does not treat mutable and immutable types differently during assignment and parameter passing. You simply cannot mutate immutable types because they have no mutating methods on them.

Original Thread

By anonymous    2017-09-20

As @DeepSpace mentioned in the comments, Ned Batchelder does a great job demystifying variables (names) and assignments to values in a blog, from which he delivered a talk at PyCon 2015, Facts and Myths about Python names and values. It can be insightful for Pythonistas at any level of mastery.

In addition, take a look at a popular SO documentation related to this topic, Creating variables and assigning values. Here you will find clear and thorough examples in Python you may find helpful.

Original Thread

By anonymous    2017-09-23

The docs carefully avoid those terms, because they are used inconsistently across communities. As far as I know, the same call mechanism is called "call by value" by Java people and "call by reference" by Ruby people.

I think it was Ned Batchelder who proposed the term "call by assignment" - because that's what implicitly happens when you call a function (edit: in Python!).

Consider:

def foo(x, y):
    return x + y

a = [1, 2, 3]
b = [4, 5, 6]
foo(a, b)

Implicitly, foo does the following before executing the rest of the function's body:

x = a
y = b

which are two assignments. So once you understand assignments - most importantly that assignment never copies data - you understand function calls.

Relevant links:
Facts and Myths about Python Names and Values (video)
Facts and Myths about Python Names and Values (html)

Original Thread

By anonymous    2017-11-20

Assignment never copies data. If you have a function foo that returns a value, then an assignment like result = foo(arg) never copies any data. (You could, of course, have copy-operations in the function's body.) Likewise, return x does not copy the object x.

Your question lacks a specific example, so I can't go into more detail.

edit: You should probably watch the excellent Facts and Myths about Python names and values talk.

Original Thread

By anonymous    2017-12-04

Ned explains everything your need to know [here](https://www.youtube.com/watch?v=_AEJHKGk9ns) perfectly.

Original Thread

By anonymous    2017-12-04

From Ned Batchelder - Facts and Myths about Python names and values - PyCon 2015:

Functions like x.append("something") and x.remove("something") mutate a value, and change their values. However, using x=x+["something"] or x=x[:-1] rebind a reference, and create a new value which the variable now points to.

Thanks to @timgeb for commenting the video!

Original Thread

By anonymous    2018-01-07

twos.append(ones) does not copy ones.

There is only ever one list ones in memory, which also goes by the following references:

  1. terms[0]
  2. twos[0]

and also terms[-1] and twos[-1] because terms and twos only have one element each, so the first is the last.

Now, when you mutate ones/terms[0]/terms[-1]/twos[0]/twos[-1] you are mutating the same list in memory.

I highly recommend watching Facts and Myths about Python names and values.

Original Thread

By anonymous    2018-03-26

What's the proper dupe target for [Facts and Myths about Python Names and Values](https://www.youtube.com/watch?v=_AEJHKGk9ns)?

Original Thread

Popular Videos 786

David Heinemeier Hansson

Submit Your Video

If you have some great dev videos to share, please fill out this form.