Using python how can I pass data between child and main processes with Queue?
Desired goal
pass information to other sub processes or main process
Concerns
-Is it a problem that the processes never finish?
-Is it a problem the class methods never return/finish?
will this cause stack overflow? ( I expect to run for hours).
Alternative
Would it be appropriate to have several different scripts and save to, load from a database/pickle?
What works
If I swap the multiprocessing for threading the desired output is achieved.
Have tried
-Tried switching the multiprocessing for threading which seems to give me the desired outcome, but would like to understand why multiprocessing is not giving the same output.
-Tried using Queue() instead or LifoQueue()
-Tried running and printing Queue.get() from another child subprocess instead of the main script
-Tried .join() all tasks
Desired output
I expect the color to be changed within a subprocess and passed to the Queue, I expect to retrieve the new color with Queue.get().
Example
import multiprocessing
import threading
import time
class MyClass:
def __init__(self):
self.color = 'gray'
def task_blue(self,q):
print('sleeping 5..')
time.sleep(5)
while True:
time.sleep(1)
self.color = 'blue'
q.put([self.color])
def task_red(self,q):
print('sleeping 5..')
time.sleep(5)
while True:
time.sleep(3)
self.color = 'red'
q.put([self.color])
def printer(self,q):
while True:
time.sleep(.1)
if q.empty():
print('<empty>')
else:
print(q.get())
from queue import LifoQueue, Queue
q = LifoQueue()
my_class = MyClass()
p1 = multiprocessing.Process(target=my_class.task_blue,args=(q,),name='BLUE')
p2 = multiprocessing.Process(target=my_class.task_red,args=(q,),name='RED')
# p3 = multiprocessing.Process(target=my_class.printer,args=(q,),name='PRINTER')
tasks = []
tasks.append(p1)
tasks.append(p2)
# tasks.append(p3)
for task in tasks:
task.start()
while True:
time.sleep(.2)
if q.empty():
print(['empty'])
else:
print(q.get())
output
Is it a problem that the processes never finish?
It's OK for a process to run forever.
Is it a problem the class methods never return/finish?
No. If you want your program to run forever, there is going to be some section of code that doesn't return. It's OK if this area of code is inside a method.
I like the idea of the entire program being one script/class
You can do it many ways. There's no correct answer here, especially absent other requirements.
Tried using Queue() instead or LifoQueue() -Tried running and printing Queue.get() from another child subprocess instead of the main script -Tried .join() all the tasks
when using the multiprocessing
package, you should use the Queue
class from the multiprocessing package. E.g. queue = multiprocessing.Queue()
. Docs here: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue