Shared variable in python's multiprocessing
When you use Value
you get a ctypes
object in shared memory that by default is synchronized using RLock
. When you use Manager
you get a SynManager
object that controls a server process which allows object values to be manipulated by other processes. You can create multiple proxies using the same manager; there is no need to create a new manager in your loop:
manager = Manager()
for i in range(5):
new_value = manager.Value('i', 0)
The Manager
can be shared across computers, while Value
is limited to one computer. Value
will be faster (run the below code to see), so I think you should use that unless you need to support arbitrary objects or access them over a network.
import time
from multiprocessing import Process, Manager, Value
def foo(data, name=''):
print type(data), data.value, name
data.value += 1
if __name__ == "__main__":
manager = Manager()
x = manager.Value('i', 0)
y = Value('i', 0)
for i in range(5):
Process(target=foo, args=(x, 'x')).start()
Process(target=foo, args=(y, 'y')).start()
print 'Before waiting: '
print 'x = {0}'.format(x.value)
print 'y = {0}'.format(y.value)
time.sleep(5.0)
print 'After waiting: '
print 'x = {0}'.format(x.value)
print 'y = {0}'.format(y.value)
To summarize:
- Use
Manager
to create multiple shared objects, including dicts and lists. UseManager
to share data across computers on a network. - Use
Value
orArray
when it is not necessary to share information across a network and the types inctypes
are sufficient for your needs. -
Value
is faster thanManager
.
Warning
By the way, sharing data across processes/threads should be avoided if possible. The code above will probably run as expected, but increase the time it takes to execute foo
and things will get weird. Compare the above with:
def foo(data, name=''):
print type(data), data.value, name
for j in range(1000):
data.value += 1
You'll need a Lock
to make this work correctly.
I am not especially knowledgable about all of this, so maybe someone else will come along and offer more insight. I figured I would contribute an answer since the question was not getting attention. Hope that helps a little.