How can I print and display subprocess stdout and stderr output without distortion?
Solution 1:
Make the pipes non-blocking by using fcntl.fcntl
, and use select.select
to wait for data to become available in either pipe. For example:
# Helper function to add the O_NONBLOCK flag to a file descriptor
def make_async(fd):
fcntl.fcntl(fd, fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK)
# Helper function to read some data from a file descriptor, ignoring EAGAIN errors
def read_async(fd):
try:
return fd.read()
except IOError, e:
if e.errno != errno.EAGAIN:
raise e
else:
return ''
process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
make_async(process.stdout)
make_async(process.stderr)
stdout = str()
stderr = str()
returnCode = None
while True:
# Wait for data to become available
select.select([process.stdout, process.stderr], [], [])
# Try reading some data from each
stdoutPiece = read_async(process.stdout)
stderrPiece = read_async(process.stderr)
if stdoutPiece:
print stdoutPiece,
if stderrPiece:
print stderrPiece,
stdout += stdoutPiece
stderr += stderrPiece
returnCode = process.poll()
if returnCode != None:
return (returnCode, stdout, stderr)
Note that fcntl
is only available on Unix-like platforms, including Cygwin.
If you need it to work on Windows without Cygwin, it's doable, but it's much, much tougher. You'll have to:
- Use the pywin32 library to call through to the native Win32 API
- Use
SetNamedPipeHandleState
withPIPE_NOWAIT
to make the stdout and stderr pipes non-blocking - Use
WaitForMultipleObjects
instead ofselect
to wait for data to become available - Use
ReadFile
to read the data