Appending to the same list from different processes using multiprocessing
I need to append objects to one list L
from different processes using multiprocessing , but it returns empty list.
How can I let many processes append to list L
using multiprocessing?
#!/usr/bin/python
from multiprocessing import Process
L=[]
def dothing(i,j):
L.append("anything")
print i
if __name__ == "__main__":
processes=[]
for i in range(5):
p=Process(target=dothing,args=(i,None))
p.start()
processes.append(p)
for p in processes:
p.join()
print L
Solution 1:
Global variables are not shared between processes.
You need to use multiprocessing.Manager.list
:
from multiprocessing import Process, Manager
def dothing(L, i): # the managed list `L` passed explicitly.
L.append("anything")
if __name__ == "__main__":
with Manager() as manager:
L = manager.list() # <-- can be shared between processes.
processes = []
for i in range(5):
p = Process(target=dothing, args=(L,i)) # Passing the list
p.start()
processes.append(p)
for p in processes:
p.join()
print L
See Sharing state between processes¶ (Server process part).
Solution 2:
Falsetru's answer worked.
But still, the list was not accessible beyond the with Manager() as manager:
two changes were needed:
-
adding
L = []
in front of theif __name__ == "__main__":
statement. Must be added as for some reason the lastprint(L)
(the one outside ofif
) is executedProcesses + 1
times. This returns an error thatL is not defined
and the code breaks. -
adding
L = list(L)
after thep.join()
statement. This step is needed to changeManager.list
to regularPython.list
- otherwise calls to themanager.list
return errors that object not readable.
from multiprocessing import Process, Manager
def dothing(L, i): # the managed list `L` passed explicitly.
for j in range(5):
text = "Process " + str(i) + ", Element " + str(j)
L.append(text)
L = []
if __name__ == "__main__":
with Manager() as manager:
L = manager.list() # <-- can be shared between processes.
processes = []
for i in range(5):
p = Process(target=dothing, args=(L,i,)) # Passing the list
p.start()
processes.append(p)
for p in processes:
p.join()
L = list(L)
print("Within WITH")
print(L)
print("Within IF")
print(L)
print("Outside of IF")
print(L)
Output:
Outside of IF
[]
Outside of IF
[]
Outside of IF
[]
Outside of IF
[]
Outside of IF
[]
Outside of IF
[]
Within WITH
['Process 2, Element 0','Process 2, Element 1', 'Process 2, Element 2',
'Process 2, Element 3', 'Process 2, Element 4', 'Process 1, Element 0',
'Process 1, Element 1', 'Process 1, Element 2', 'Process 1, Element 3',
'Process 1, Element 4', 'Process 0, Element 0', 'Process 0, Element 1',
'Process 0, Element 2', 'Process 0, Element 3', 'Process 0, Element 4',
'Process 4, Element 0', 'Process 4, Element 1', 'Process 4, Element 2',
'Process 4, Element 3', 'Process 4, Element 4', 'Process 3, Element 0',
'Process 3, Element 1', 'Process 3, Element 2', 'Process 3, Element 3',
'Process 3, Element 4']
Within IF
['Process 2, Element 0','Process 2, Element 1', 'Process 2, Element 2',
'Process 2, Element 3', 'Process 2, Element 4', 'Process 1, Element 0',
'Process 1, Element 1', 'Process 1, Element 2', 'Process 1, Element 3',
'Process 1, Element 4', 'Process 0, Element 0', 'Process 0, Element 1',
'Process 0, Element 2', 'Process 0, Element 3', 'Process 0, Element 4',
'Process 4, Element 0', 'Process 4, Element 1', 'Process 4, Element 2',
'Process 4, Element 3', 'Process 4, Element 4', 'Process 3, Element 0',
'Process 3, Element 1', 'Process 3, Element 2', 'Process 3, Element 3',
'Process 3, Element 4']
Outside of IF
['Process 2, Element 0','Process 2, Element 1', 'Process 2, Element 2',
'Process 2, Element 3', 'Process 2, Element 4', 'Process 1, Element 0',
'Process 1, Element 1', 'Process 1, Element 2', 'Process 1, Element 3',
'Process 1, Element 4', 'Process 0, Element 0', 'Process 0, Element 1',
'Process 0, Element 2', 'Process 0, Element 3', 'Process 0, Element 4',
'Process 4, Element 0', 'Process 4, Element 1', 'Process 4, Element 2',
'Process 4, Element 3', 'Process 4, Element 4', 'Process 3, Element 0',
'Process 3, Element 1', 'Process 3, Element 2', 'Process 3, Element 3',
'Process 3, Element 4']
Solution 3:
Thanks to @falsetru for suggesting the exact documentation and providing the good code. I need to keep the order for my application and by modifying the @falsetru code, now the below code preserves the order of adding items to the list.
The sleep is helpful to catch the bugs otherwise it is hard to catch the problem with ordering of the list.
from multiprocessing import Process, Manager
from time import sleep
def dothing(L, i): # the managed list `L` passed explicitly.
L[i]= i
sleep(4)
if __name__ == "__main__":
with Manager() as manager:
L = manager.list(range(50)) # <-- can be shared between processes.
processes = []
for i in range(50):
p = Process(target=dothing, args=(L,i)) # Passing the list
p.start()
processes.append(p)
for p in processes:
p.join()
print(L)