Python Multiprocessing Process or Pool for what I am doing?

Solution 1:

The two scenarios you listed accomplish the same thing but in slightly different ways.

The first scenario starts two separate processes (call them P1 and P2) and starts P1 running foo and P2 running bar, and then waits until both processes have finished their respective tasks.

The second scenario starts two processes (call them Q1 and Q2) and first starts foo on either Q1 or Q2, and then starts bar on either Q1 or Q2. Then the code waits until both function calls have returned.

So the net result is actually the same, but in the first case you're guaranteed to run foo and bar on different processes.

As for the specific questions you had about concurrency, the .join() method on a Process does indeed block until the process has finished, but because you called .start() on both P1 and P2 (in your first scenario) before joining, then both processes will run asynchronously. The interpreter will, however, wait until P1 finishes before attempting to wait for P2 to finish.

For your questions about the pool scenario, you should technically use pool.close() but it kind of depends on what you might need it for afterwards (if it just goes out of scope then you don't need to close it necessarily). pool.map() is a completely different kind of animal, because it distributes a bunch of arguments to the same function (asynchronously), across the pool processes, and then waits until all function calls have completed before returning the list of results.

Solution 2:

Since you're fetching data from curl calls you are IO-bound. In such case grequests might come in handy. These are really neither processes nor threads but coroutines - lightweight threads. This would allow you to send asynchronously HTTP requests, and then use multiprocessing.Pool to speed up the CPU-bound part.

1) Since join blocks until calling process is completed...does this mean p1 process has to finish before p2 process is kicked off?

Yes, p2.join() is called after p1.join() has returned meaning p1 has finished.

1) Do I need a pool.close(), pool.join()

You could end up with orphaned processes without doing close() and join() (if the processes serve indefinetly)

2) Would pool.map() make them all complete before I could get results? And if so, are they still ran asynch?

They are ran asynchronously, but the map() is blocked until all tasks are done.

3) How would pool.apply_async() differ from doing each process with pool.apply()

pool.apply() is blocking, so basically you would do the processing synchronously.

4) How would this differ from the previous implementation with Process

Chances are a worker is done with foo before you apply bar so you might end up with a single worker doing all the work. Also, if one of your workers dies Pool automatically spawns a new one (you'd need to reapply the task).

To sum up: I would rather go with Pool - it's perfect for producer-consumer cases and takes care of all the task-distributing logic.