How do I get 'real-time' information back from a subprocess.Popen in python (2.5)

Update with code that appears not to work (on windows anyway)

class ThreadWorker(threading.Thread):
    def __init__(self, callable, *args, **kwargs):
        super(ThreadWorker, self).__init__()
        self.callable = callable
        self.args = args
        self.kwargs = kwargs
        self.setDaemon(True)

    def run(self):
        try:
            self.callable(*self.args, **self.kwargs)
        except wx.PyDeadObjectError:
            pass
        except Exception, e:
            print e



if __name__ == "__main__":
    import os
    from subprocess import Popen, PIPE

    def worker(pipe):
        while True:
            line = pipe.readline()
            if line == '': break
            else: print line

    proc = Popen("python subprocess_test.py", shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)

    stdout_worker = ThreadWorker(worker, proc.stdout)
    stderr_worker = ThreadWorker(worker, proc.stderr)
    stdout_worker.start()
    stderr_worker.start()
    while True: pass

stdout will be buffered - so you won't get anything till that buffer is filled, or the subprocess exits.

You can try flushing stdout from the sub-process, or using stderr, or changing stdout on non-buffered mode.

It sounds like the issue might be the use of buffered output by the subprocess - if a relatively small amount of output is created, it could be buffered until the subprocess exits. Some background can be found here:

Here's what worked for me:

cmd = ["./tester_script.bash"]
p = subprocess.Popen( cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE )
while p.poll() is None:
    out = p.stdout.readline()
    do_something_with( out, err )

In your case you could try to pass a reference to the sub-process to your Worker Thread, and do the polling inside the thread. I don't know how it will behave when two threads poll (and interact with) the same subprocess, but it may work.

Also note thate the while p.poll() is None: is intended as is. Do not replace it with while not p.poll() as in python 0 (the returncode for successful termination) is also considered False.