Is there a better way of doing multiprocessing than this? Preferably faster and not so hard on the memory

This is my code. I don't think this is the best way to do this task, Can you kindly help me? Thanks in advance. It responded once but when I ran it again it kept on running and didn't give out the desired output. Edit: I have added the missing code, platform wasn't allowing me to post it as there is too much code and too little description.

Q=Queue()
Q1=Queue()
Q2=Queue()
Q3=Queue()
Q4=Queue()
Q5=Queue()
Q6=Queue()
Q7=Queue()
Q8=Queue()
p0=  Process(target=weatherSearch,args=(city,Q,))
p2=  Process(target=holidays,args=(city,Q2,))
p3=  Process(target=constructUniversity,args=(school,company,Q3,))
p4=  Process(target=constructsignature,args=(Q4,))
p5=  Process(target=constructCTA,args=(Q5,))
p6=  Process(target=profile,args=(df,Q6,))
p7=  Process(target=google_news,args=(company,Q7,))
p8=  Process(target=financial_news,args=(company,Q8,))






p0.start()
p2.start()
p3.start()
p4.start()
p5.start()
p6.start()
p7.start()
p8.start()

p0.join()
p2.join()
p3.join()
p4.join()
p5.join()
p6.join()
p7.join()
p8.join()
result=Q.get()
temperature=result["temp"][0]
conditions=result["weather_description"][0]
holidays=Q2.get()
constructuniversity=Q3.get()
signature=Q4.get()
CTA=Q5.get()
general_news=Q7.get()
finance=Q8.get()
resul=Q6.get()
myers_briggs=resul[0]
rules=resul[1]
better_understanding=resul[2]
personality_type=resul[3]

Solution 1:

I want to preface this by saying, your solution doesn't necessarily have any problems with it as long as it works as you expect it to.

I would re-structure your functions to simply return their results rather than putting the results each to their own Queue, then use a multiprocessing.Pool to call the functions (because it will handle passing args and results back and forth for you). I would also get in the habit of using the if __name__ == "__main__": clause in case you aren't already. It makes your code less likely to fail if you try to run it on another computer.

if __name__ == "__main__":
    #in general don't create processes outside this if statement [calling Pool() creates processes as well as simply Process()]
    #This will make your code more compatible with Windows and MacOS (with python later than 3.6 iirc)
    with multiprocessing.Pool() as pool:
        #Call functions asynchronously then call .get() on the async result to get the return value when it becomes available.
        result_async = pool.apply_async(weatherSearch, (city,))
        holidays_async = pool.apply_async(holidays, (city,))
        constructuniversity_async = pool.apply_async(constructUniversity, (school, company))
        signature_async = pool.apply_async(constructsignature)
        CTA_async = pool.apply_async(constructCTA)
        general_news_async = pool.apply_async(google_news, (company,))
        finance_async = pool.apply_async(financial_news, (company,))
        resul_async = pool.apply_async(profile, (df,))
        
        result = result_async.get()
        temperature=result["temp"][0]
        conditions=result["weather_description"][0]
        holidays=holidays_async.get()
        constructuniversity=constructuniversity_async.get()
        signature=signature_async.get()
        CTA=CTA_async.get()
        general_news=general_news_async.get()
        finance=finance_async.get()
        resul=resul_async.get()
        myers_briggs=resul[0]
        rules=resul[1]
        better_understanding=resul[2]
        personality_type=resul[3]

The usage is a little cleaner than creating a bunch of unique Queue's, and I would stick to more descriptive names than things like Q1, Q2, Q3, etc.. Finally I would like to point out, that if you stick with your original solution, you should "get" the results from the Queue's before "joining" the processes. It's extremely unlikely to happen with how you're using them, but in theory this could cause a process to get stuck and never shut down (and then the main process would be stuck waiting for join). In practice the fix is as simple as moving all the join statements to the bottom.