David Beazley - Python Concurrency From the Ground Up: LIVE! - PyCon 2015

By: PyCon 2015

1338   6   89937

Uploaded on 04/11/2015

"Speaker: David Beazley

There are currently three popular approaches to Python concurrency: threads, event loops, and coroutines. Each is shrouded by various degrees of mystery and peril. In this talk, all three approaches will be deconstructed and explained in a epic ground-up live coding battle.

Slides can be found at: https://speakerdeck.com/pycon2015 and https://github.com/PyCon/2015-slides"

Comments (7):

By anonymous    2017-09-20

Short answer:

yield from is an old way to await for coroutine.

await is an modern way to await for coroutine.

Detailed answer:

Python has generators - special kind of functions that produces a sequence of results instead of a single value. Starting with Python 3.3 yield from expression was added. It allows one generator to delegate part of its operations to another generator.

Starting with Python 3.4 asyncio module was added to standard library. It allow us to write clear and understandable asynchronous code. While technically asyncio's coroutines could be implemented different ways, in asyncio they were implemented using generators (you can watch for excellent video where shown how generators can be used to implement coroutines). @asyncio.coroutine was a way to make coroutine from generator and yield from was a way to await for coroutine - just details of implementation.

That's how happened that yield from started to be used for two "different things".

Starting with Python 3.5 (see PEP 492) coroutines got need syntax. Now you can define coroutine with async def and await for it using await expression. It's not only shorter to write, but also makes clearer to understand that we work with coroutines.

If you're using Python 3.5+ you can forgot of using yield from for anything except generators and use await for coroutines.

Original Thread

By anonymous    2017-09-20

Does each task get launched on separate thread?

No, usually asyncio runs in single thread.

How asyncio package enables bar, to be an async task, with these keywords, under the hood?

When you define function as async this function becomes generator what allows to execute it "by steps" using __next__() method. await - is yield (yield from actually) point where execution flow returns to global event loop that manages executing of all coroutines.

This simple example shows how you can switch between execution flow of different generators:

def task(i):
    yield 1
    print('task {}: step 1'.format(i))
    yield 2
    print('task {}: step 2'.format(i))


tasks = [
    task(1),
    task(2),
    task(3),
]


def execute_tasks(tasks):
    i = 0
    finished = []
    while True:
        # start executing tasks:
        try:
            tasks[i].__next__()
        except StopIteration:
            finished.append(i)
        # check if any task unfinished:
        if len(finished) == len(tasks):
            return
        # move to next unfinished task:
        while True:
            i += 1
            if i > len(tasks) - 1:
                i = 0
            if not i in finished:
                break


if __name__ == '__main__':
    execute_tasks(tasks)

Output:

task 1: step 1
task 2: step 1
task 3: step 1
task 1: step 2
task 2: step 2
task 3: step 2

asyncio of course is much more complex and allows you much more.

Probably best explanation of how you can implement coroutines using generators I saw in this PyCon 2015 video: David Beazley - Python Concurrency From the Ground Up: LIVE! (source code). You should definitely watch it if you're going implement this.

But I advice you to use asyncio instead - it already exists for you, there's no need to invent your own.

Original Thread

By anonymous    2017-10-22

@wvxvw I really think it contains all needed stuff to see how to use asyncio. If you want something more deep, to understand how asyncio itself works, I can advice you to watch this video: https://www.youtube.com/watch?v=MCs5OvhV9S4 It's more complex, but it shows how event loop (abstract one) organized and how it manages things.

Original Thread

By anonymous    2018-01-07

Code you provided is not concurrent in both cases. It happens because of this line:

async def asleep(n):
    time.sleep(n)   # blocking method

time.sleep freezes asyncio event loop making it unable to process other running same time coroutines. You should never do such thing when using asyncio (if you want to run some blocking code inside coroutine, you should run it in another thread using loop.run_in_executor as shown here).

To make your code coroutine working properly you should use special asyncio version of sleep that doesn't block event loop:

async def asleep(n):
    await asyncio.sleep(n)   # blocking method

Compare executing this version (in case1) with version you posted (in case1): 6 seconds vs. 2 seconds.


Once we creating tasks from coroutines using ensure_future these coroutines start running concurently:

tasks = [asyncio.ensure_future(c) for c in coros]  # this run all coroutines concurently

Using asyncio.gather - is alternative way to run coroutines concurently:

await asyncio.gather(*coros)  # this run all coroutines concurently

Way asyncio works in detail may seem pretty complex. Best explanation I've seen: David Beazley - Python Concurrency From the Ground Up. Note, that you usually don't need to understand low-level detail to use asyncio on practice.

Original Thread

By anonymous    2018-01-07

This seems like a pretty straightforward use case for asyncio. I wouldn't consider using asyncio as "using a particular library" since socket programming paired with asyncio's event loop is pretty low-level and the concept is very transparent if you have experience with other languages and just want to see how async programming works in Python.

You can use this async chat as an example: https://gist.github.com/gregvish/7665915

Essentially, you create a non-blocking socket, see standard library reference on socket.setblocking(0):

https://docs.python.org/3/library/socket.html#socket.socket.setblocking

I'd also suggest this amazing session by David Beazley as a must-see for async Python programming. He explains the concurrency concepts in Python using sockets, exactly what you need: https://www.youtube.com/watch?v=MCs5OvhV9S4

Original Thread

Recommended Books

    Submit Your Video

    If you have some great dev videos to share, please fill out this form.