Thinking about Concurrency, Raymond Hettinger, Python core developer

By: Василий Айтипиплов

616   9   38262

Uploaded on 07/13/2016

Comments (8):

By anonymous    2017-09-20

Never try to kill a thread from something external to that thread. You never know if that thread is holding a lock. Python doesn’t provide a direct mechanism for kill threads externally; however, you can do it using ctypes, but that is a recipe for a deadlock.

This quote is from Raymond Hettinger, there is a speech about this.

Original Thread

By anonymous    2017-09-20

Raymond Hettinger gave a really good keynote talk about the proper way to think about concurrency and multithreading here: https://www.youtube.com/watch?v=Bv25Dwe84g0&t=2

And his notes can be found here: https://dl.dropboxusercontent.com/u/3967849/pyru/_build/html/index.html

What I recommend, which is from the talk, is to use an atomic message queue to "talk" between the threads. However, this talk and Raymond's work is done in 3.5 or 3.6. This library https://docs.python.org/3/library/queue.html will help you significantly.

Original Thread

By anonymous    2017-09-20

No, Python does have multithreading. In fact, it uses system threads. The problem is just that it can't use more than one of the available cores. This is due to something called the GIL(Global Interpreter Lock). Python threads still work for I/O bound tasks as opposed to CPU bound tasks which may cause deadlocks and race conditions. Many Python libraries solve this issue by using C extensons to bypass the GIL. Of course, this is all in the case of CPython.

There is a very interesting talk about this by one of the core developers of Python.

Thinking about concurrency, Raymond Hettinger

Now you are right, it is much better to use multiprocessing to get the benefit of all the cores. But there are much fewer cores than there are threads. Cores are valuable resources and take up a lot of memory. If you don't mind dealing with IPC(Interprocess Communication), then it is a great solution.

Original Thread

By anonymous    2018-02-05

I can't give that question justice. The answer is moderately nuanced. This talk by Raymond Hettinger might be a good start: https://www.youtube.com/watch?v=Bv25Dwe84g0

Original Thread

By anonymous    2018-05-09

As @lutz had written, there is no danger of race conditions in case there are no shared data between two instances of the common_function. This is what is meant by referential transparency and generally all programming languages aim that such functions should be thread safe. Sometimes, you will need to write and use functions which change some global state. In such cases the modern adage is to use event driven programming - which means to not communicate directly between threads but communicate via some thread safe queueing system. In python I am a huge fan of the queue module for that matter. Another good queue module is the multiprocessing.queue for which a good example is here. I am also pasting the code here.

from multiprocessing import Process, Queue

def f(q):
    q.put([42, None, 'hello'])

if __name__ == '__main__':
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    print(q.get())    # prints "[42, None, 'hello']"
    p.join()

Finally in case you are not confident about some function (probably it is a big function and you dont understand the nuts and bolts), I would recommend that you use the fuzzying method. Here you define a simple function

FUZZ = True
def fuzz():
    """
    fuzzing is a technique to make the race condition errors more visible
    """
    if FUZZ:
        time.sleep(random.random())

and then drop this function at random places inside your code. That should amplify any race conditions that there are in the code. This is of-course not a guaranteed method and so if your function is in a production application that gets called millions of times the better strategy will be to break the function to smaller more digestible parts. Watch Raymond Hettinger deliver a lecture about concurrent code in his famous talk on python threading. You can get the code that he is talking about here.

Original Thread

Submit Your Video

If you have some great dev videos to share, please fill out this form.