Log output of multiprocessing.Process

Solution 1:

The easiest way might be to just override sys.stdout. Slightly modifying an example from the multiprocessing manual:

from multiprocessing import Process
import os
import sys

def info(title):
    print title
    print 'module name:', __name__
    print 'parent process:', os.getppid()
    print 'process id:', os.getpid()

def f(name):
    sys.stdout = open(str(os.getpid()) + ".out", "w")
    info('function f')
    print 'hello', name

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    q = Process(target=f, args=('fred',))
    q.start()
    p.join()
    q.join()

And running it:

$ ls
m.py
$ python m.py
$ ls
27493.out  27494.out  m.py
$ cat 27493.out 
function f
module name: __main__
parent process: 27492
process id: 27493
hello bob
$ cat 27494.out 
function f
module name: __main__
parent process: 27492
process id: 27494
hello fred

Solution 2:

There are only two things I would add to @Mark Rushakoff answer. When debugging, I found it really useful to change the buffering parameter of my open() calls to 0.

sys.stdout = open(str(os.getpid()) + ".out", "a", buffering=0)

Otherwise, madness, because when tail -fing the output file the results can be verrry intermittent. buffering=0 for tail -fing great.

And for completeness, do yourself a favor and redirect sys.stderr as well.

sys.stderr = open(str(os.getpid()) + "_error.out", "a", buffering=0)

Also, for convenience you might dump that into a separate process class if you wish,

class MyProc(Process):
    def run(self):
        # Define the logging in run(), MyProc's entry function when it is .start()-ed 
        #     p = MyProc()
        #     p.start()
        self.initialize_logging()

        print 'Now output is captured.'

        # Now do stuff...

    def initialize_logging(self):
        sys.stdout = open(str(os.getpid()) + ".out", "a", buffering=0)
        sys.stderr = open(str(os.getpid()) + "_error.out", "a", buffering=0)

        print 'stdout initialized'

Heres a corresponding gist

Solution 3:

You can set sys.stdout = Logger() where Logger is a class whose write method (immediately, or accumulating until a \n is detected) calls logging.info (or any other way you want to log). An example of this in action.

I'm not sure what you mean by "a given" process (who's given it, what distinguishes it from all others...?), but if you mean you know what process you want to single out that way at the time you instantiate it, then you could wrap its target function (and that only) -- or the run method you're overriding in a Process subclass -- into a wrapper that performs this sys.stdout "redirection" -- and leave other processes alone.

Maybe if you nail down the specs a bit I can help in more detail...?

Solution 4:

Here is the simple and straightforward way for capturing stdout for multiprocessing.Process and io.TextIOWrapper:

import app
import io
import sys
from multiprocessing import Process


def run_app(some_param):
    out_file = open(sys.stdout.fileno(), 'wb', 0)
    sys.stdout = io.TextIOWrapper(out_file, write_through=True)
    app.run()

app_process = Process(target=run_app, args=('some_param',))
app_process.start()
# Use app_process.termninate() for python <= 3.7.
app_process.kill()