Are Python built-in containers thread-safe?

I would like to know if the Python built-in containers (list, vector, set...) are thread-safe? Or do I need to implement a locking/unlocking environment for my shared variable?

Solution 1:

You need to implement your own locking for all shared variables that will be modified in Python. You don't have to worry about reading from the variables that won't be modified (ie, concurrent reads are ok), so immutable types (frozenset, tuple, str) are probably safe, but it wouldn't hurt. For things you're going to be changing - list, set, dict, and most other objects, you should have your own locking mechanism (while in-place operations are ok on most of these, threads can lead to super-nasty bugs - you might as well implement locking, it's pretty easy).

By the way, I don't know if you know this, but locking is very easy in Python - create a threading.lock object, and then you can acquire/release it like this:

import threading
list1Lock = threading.Lock()

with list1Lock:
    # change or read from the list here
# continue doing other stuff (the lock is released when you leave the with block)

In Python 2.5, do from __future__ import with_statement; Python 2.4 and before don't have this, so you'll want to put the acquire()/release() calls in try:...finally: blocks:

import threading
list1Lock = threading.Lock()

try:
    list1Lock.acquire()
    # change or read from the list here
finally:
    list1Lock.release()
# continue doing other stuff (the lock is released when you leave the with block)

Some very good information about thread synchronization in Python.

Solution 2:

Yes, but you still need to be careful of course

For example:

If two threads are racing to pop() from a list with only one item, One thread will get the item successfully and the other will get an IndexError

Code like this is not thread-safe

if L:
    item=L.pop() # L might be empty by the time this line gets executed

You should write it like this

try:
    item=L.pop()
except IndexError:
    # No items left