Raymond Hettinger Modern Python Dictionaries A confluence of a dozen great ideas PyCon 2017

By: PyCon 2017

385   17   25695

Uploaded on 05/20/2017

"Speaker: Raymond Hettinger

Python's dictionaries are stunningly good. Over the years,
many great ideas have combined together to produce the
modern implementation in Python 3.6.

This fun talk uses pictures and little bits of pure python
code to explain all of the key ideas and how they evolved
over time.

Includes newer features such as key-sharing, compaction, and versioning.

Slides can be found at: https://speakerdeck.com/pycon2017 and https://github.com/PyCon/2017-slides"

Comments (14):

By emj    2017-09-20

YT video: https://www.youtube.com/watch?v=npw4s1QTmPg slides for the 2016 talk: https://dl.dropboxusercontent.com/u/3967849/sfmu2/_build/htm...

The 2017 slides are better, but I can't find them they are mentioned in the beginning of the talk though.

Original Thread

By sidmitra    2017-09-20

Raymond Hettingers presentation this year at Pycon was also interesting.

Modern Python Dictionaries A confluence of a dozen great ideas PyCon 2017 - https://www.youtube.com/watch?v=npw4s1QTmPg

Original Thread

By hprotagonist    2017-09-20

hettinger's updates at pycon this year: it's better! https://www.youtube.com/watch?v=npw4s1QTmPg

Original Thread

By vonseel    2018-02-07

There’s a well known YouTube video which covers all this comprehensively.

Raymond Hettinger, Modern Python Dictionaries... hopefully this is the correct one. https://youtu.be/npw4s1QTmPg

Original Thread

By anonymous    2017-09-20

How can I merge two Python dictionaries in a single expression?

Say you have two dicts and you want to merge them into a new dict without altering the original dicts:

x = {'a': 1, 'b': 2}
y = {'b': 3, 'c': 4}

The desired result is to get a new dictionary (z) with the values merged, and the second dict's values overwriting those from the first.

>>> z
{'a': 1, 'b': 3, 'c': 4}

A new syntax for this, proposed in PEP 448 and available as of Python 3.5, is

z = {**x, **y}

And it is indeed a single expression. It is now showing as implemented in the release schedule for 3.5, PEP 478, and it has now made its way into What's New in Python 3.5 document.

However, since many organizations are still on Python 2, you may wish to do this in a backwards compatible way. The classically Pythonic way, available in Python 2 and Python 3.0-3.4, is to do this as a two-step process:

z = x.copy()
z.update(y) # which returns None since it mutates z

In both approaches, y will come second and its values will replace x's values, thus 'b' will point to 3 in our final result.

Not yet on Python 3.5, but want a single expression

If you are not yet on Python 3.5, or need to write backward-compatible code, and you want this in a single expression, the most performant while correct approach is to put it in a function:

def merge_two_dicts(x, y):
    """Given two dicts, merge them into a new dict as a shallow copy."""
    z = x.copy()
    return z

and then you have a single expression:

z = merge_two_dicts(x, y)

You can also make a function to merge an undefined number of dicts, from zero to a very large number:

def merge_dicts(*dict_args):
    Given any number of dicts, shallow copy and merge into a new dict,
    precedence goes to key value pairs in latter dicts.
    result = {}
    for dictionary in dict_args:
    return result

This function will work in Python 2 and 3 for all dicts. e.g. given dicts a to g:

z = merge_dicts(a, b, c, d, e, f, g) 

and key value pairs in g will take precedence over dicts a to f, and so on.

Critiques of Other Answers

Don't use what you see in the formerly accepted answer:

z = dict(x.items() + y.items())

In Python 2, you create two lists in memory for each dict, create a third list in memory with length equal to the length of the first two put together, and then discard all three lists to create the dict. In Python 3, this will fail because you're adding two dict_items objects together, not two lists -

>>> c = dict(a.items() + b.items())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'dict_items' and 'dict_items'

and you would have to explicitly create them as lists, e.g. z = dict(list(x.items()) + list(y.items())). This is a waste of resources and computation power.

Similarly, taking the union of items() in Python 3 (viewitems() in Python 2.7) will also fail when values are unhashable objects (like lists, for example). Even if your values are hashable, since sets are semantically unordered, the behavior is undefined in regards to precedence. So don't do this:

>>> c = dict(a.items() | b.items())

This example demonstrates what happens when values are unhashable:

>>> x = {'a': []}
>>> y = {'b': []}
>>> dict(x.items() | y.items())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'

Here's an example where y should have precedence, but instead the value from x is retained due to the arbitrary order of sets:

>>> x = {'a': 2}
>>> y = {'a': 1}
>>> dict(x.items() | y.items())
{'a': 2}

Another hack you should not use:

z = dict(x, **y)

This uses the dict constructor, and is very fast and memory efficient (even slightly more-so than our two-step process) but unless you know precisely what is happening here (that is, the second dict is being passed as keyword arguments to the dict constructor), it's difficult to read, it's not the intended usage, and so it is not Pythonic.

Here's an example of the usage being remediated in django.

Dicts are intended to take hashable keys (e.g. frozensets or tuples), but this method fails in Python 3 when keys are not strings.

>>> c = dict(a, **b)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: keyword arguments must be strings

From the mailing list, Guido van Rossum, the creator of the language, wrote:

I am fine with declaring dict({}, **{1:3}) illegal, since after all it is abuse of the ** mechanism.


Apparently dict(x, **y) is going around as "cool hack" for "call x.update(y) and return x". Personally I find it more despicable than cool.

It is my understanding (as well as the understanding of the creator of the language) that the intended usage for dict(**y) is for creating dicts for readability purposes, e.g.:

dict(a=1, b=10, c=11)

instead of

{'a': 1, 'b': 10, 'c': 11}

Response to comment

Despite what Guido says, dict(x, **y) is in line with the dict specification, which btw. works for both Python 2 and 3. The fact that this only works for string keys is a direct consequence of how keyword parameters work and not a short-comming of dict. Nor is using the ** operator in this place an abuse of the mechanism, in fact ** was designed precisely to pass dicts as keywords.

Again, it doesn't work for 3 when keys are non-strings. The implicit calling contract is that namespaces take ordinary dicts, while users must only pass keyword arguments that are strings. All other callables enforced it. dict broke this consistency in Python 2:

>>> foo(**{('a', 'b'): None})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: foo() keywords must be strings
>>> dict(**{('a', 'b'): None})
{('a', 'b'): None}

This inconsistency was bad given other implementations of Python (Pypy, Jython, IronPython). Thus it was fixed in Python 3, as this usage could be a breaking change.

I submit to you that it is malicious incompetence to intentionally write code that only works in one version of a language or that only works given certain arbitrary constraints.

Less Performant But Correct Ad-hocs

These approaches are less performant, but they will provide correct behavior. They will be much less performant than copy and update or the new unpacking because they iterate through each key-value pair at a higher level of abstraction, but they do respect the order of precedence (latter dicts have precedence)

You can also chain the dicts manually inside a dict comprehension:

{k: v for d in dicts for k, v in d.items()} # iteritems in Python 2.7

or in python 2.6 (and perhaps as early as 2.4 when generator expressions were introduced):

dict((k, v) for d in dicts for k, v in d.items())

itertools.chain will chain the iterators over the key-value pairs in the correct order:

import itertools
z = dict(itertools.chain(x.iteritems(), y.iteritems()))

Performance Analysis

I'm only going to do the performance analysis of the usages known to behave correctly.

import timeit

The following is done on Ubuntu 14.04

In Python 2.7 (system Python):

>>> min(timeit.repeat(lambda: merge_two_dicts(x, y)))
>>> min(timeit.repeat(lambda: {k: v for d in (x, y) for k, v in d.items()} ))
>>> min(timeit.repeat(lambda: dict(itertools.chain(x.iteritems(), y.iteritems()))))
>>> min(timeit.repeat(lambda: dict((k, v) for d in (x, y) for k, v in d.items())))

In Python 3.5 (deadsnakes PPA):

>>> min(timeit.repeat(lambda: {**x, **y}))
>>> min(timeit.repeat(lambda: merge_two_dicts(x, y)))
>>> min(timeit.repeat(lambda: {k: v for d in (x, y) for k, v in d.items()} ))
>>> min(timeit.repeat(lambda: dict(itertools.chain(x.items(), y.items()))))
>>> min(timeit.repeat(lambda: dict((k, v) for d in (x, y) for k, v in d.items())))

Resources on Dictionaries

Original Thread

By anonymous    2017-09-20

How are Python's Built In Dictionaries Implemented?

Python's Dictionaries are Hash Tables

For a long time, it worked like this. Python would preallocate 8 empty rows and use the hash to determine where to stick the key-value pair. For example, if the hash for the key ended in 001, it would stick it in the 1 index (like the example below.)

     hash         key    value
     null        null    null
...010001    ffeb678c    633241c4 # addresses of the keys and values
     null        null    null
      ...         ...    ...

Each row takes up 24 bytes on a 64 bit architecture, 12 on a 32 bit. (Note that the column headers are just labels - they don't actually exist in memory.)

If the hash ended the same as a preexisting key's hash, this is a collision, and then it would stick the key-value pair in a different location.

After 5 key-values are stored, when adding another key-value pair, the probability of hash collisions is too large, so the dictionary is doubled in size. In a 64 bit process, before the resize, we have 72 bytes empty, and after, we are wasting 240 bytes due to the 10 empty rows.

This takes a lot of space, but the lookup time is fairly constant. The key comparison algorithm is to compute the hash, go to the expected location, compare the key's id - if they're the same object, they're equal. If not then compare the hash values, if they are not the same, they're not equal. Else, then we finally compare keys for equality, and if they are equal, return the value. The final comparison for equality can be quite slow, but the earlier checks usually shortcut the final comparison, making the lookups very quick.

(Collisions slow things down, and an attacker could theoretically use hash collisions to perform a denial of service attack, so we randomized the hash function such that it computes a different hash for each new Python process.)

The wasted space described above has led us to modify the implementation of dictionaries, with an exciting new (if unofficial) feature that dictionaries are now ordered (by insertion).

The New Compact Hash Tables

We start, instead, by preallocating an array for the index of the insertion.

Since our first key-value pair goes in the second slot, we index like this:

[null, 0, null, null, null, null, null, null]

And our table just gets populated by insertion order:

     hash         key    value
...010001    ffeb678c    633241c4 
      ...         ...    ...

So when we do a lookup for a key, we use the hash to check the position we expect (in this case, we go straight to index 1 of the array), then go to that index in the hash-table (e.g. index 0), check that the keys are equal (using the same algorithm described earlier), and if so, return the value.

We retain constant lookup time, with minor speed losses in some cases and gains in others, with the upside that we save quite a lot of space over the pre-existing implementation. The only space wasted are the null bytes in the index array.

Raymond Hettinger introduced this to python-dev in December of 2012. It finally got into CPython in Python 3.6. Ordering by insertion is still considered an implementation detail to allow other implementations of Python a chance to catch up.

Shared Keys

Another optimization to save space is an implementation that shares keys. Thus, instead of having redundant dictionaries that take up all of that space, we have dictionaries that reuse the shared keys and keys' hashes. You can think of it like this:

     hash         key    dict_0    dict_1    dict_2...
...010001    ffeb678c    633241c4  fffad420  ...
      ...         ...    ...       ...       ...

For a 64 bit machine, this could save up to 16 bytes per key per extra dictionary.

Shared Keys for Custom Objects & Alternatives

These shared-key dicts are intended to be used for custom objects' __dict__. To get this behavior, I believe you need to finish populating your __dict__ before you instantiate your next object (see PEP 412). This means you should assign all your attributes in the __init__ or __new__, else you might not get your space savings.

However, if you know all of your attributes at the time your __init__ is executed, you could also provide __slots__ for your object, and guarantee that __dict__ is not created at all (if not available in parents), or even allow __dict__ but guarantee that your foreseen attributes are stored in slots anyways. For more on __slots__, see my answer here.

See also:

Original Thread

By anonymous    2017-09-20


An overview of how Python's dictionaries are implemented can be found in the 2017 Pycon talk, Modern Python Dictionaries A confluence of a dozen great ideas. The slides can be found here.

How to visualize reduction

I understand that a hash table will use a hashing function to reduce the universe of all possible keys down to a set m and use chaining to resolve collisions. ... I can't seem to visualize the m part of it.

The easiest visualization is with m == 2 so that hashing divides keys into two groups:

>>> from pprint import pprint
>>> def hash(n):
        'Hash a number into evens or odds'
        return n % 2

>>> table = [[], []]
>>> for x in [10, 15, 12, 41, 80, 13, 40, 9]:

>>> pprint(table, width=25)
[[10, 12, 80, 40],
 [15, 41, 13, 9]]

In the above example, the eight keys all get divided into two groups (the evens and the odds).

The example also works with bigger values of m such as m == 7:

>>> table = [[], [], [], [], [], [], []]
>>> for x in [10, 15, 12, 41, 80, 13, 40, 9]:
        table[x % 7].append(x)

>>> pprint(table, width=25)
 [10, 80],
 [12, 40],
 [41, 13]]

As you can see, the above example has two empty slots and slots with a collision.

Table for an empty dict

Say I create an empty dict() in python. Does python create a table with some predefined number of null entries?

Yes, Python creates eight slots for an empty table. In Python's source code, we see #define PyDict_MINSIZE 8 in cpython/Objects/dictobject.c.

Original Thread

By anonymous    2017-09-23

If you have 30 minutes, this is a great video to watch about dicts, from one of the people involved in their implementation in python - https://www.youtube.com/watch?v=npw4s1QTmPg

Original Thread

By anonymous    2017-10-08

There is no problem with using a dictionary with list values. dictionaries are actually very efficient in python and used everywhere "under the hod" (also in classes). My first approach would be using a plain dictionary with score lists as values. If you need more flexibility later you could then switch to a custom class with additional functionality. Maybe you find this [video](https://www.youtube.com/watch?v=npw4s1QTmPg) about the topic interesting.

Original Thread

By anonymous    2017-12-04

There is a great talk about the various versions of Python dictionaries by Raymond Hetttinger here: https://www.youtube.com/watch?v=npw4s1QTmPg

Original Thread

By anonymous    2018-05-01

@FlyingTeller https://www.youtube.com/watch?v=npw4s1QTmPg

Original Thread

Popular Videos 500


Submit Your Video

If you have some great dev videos to share, please fill out this form.