Force another program's standard output to be unbuffered using Python
Solution 1:
You can use PTYs to solve this by:
- Creating a pty master/slave pair;
- Connecting the child process's stdin, stdout and stderr to the pty slave device;
- Reading from and writing to the pty master in the parent.
Solution 2:
Doing this is possible, but the only solution I can think of is fairly convoluted, non-portable, and probably fraught with problematic details. You can use LD_PRELOAD to cause the external application to load a dynamic library which contains a constructor that invokes setvbuf to unbuffer stdout. You will probably also want to wrap setvbuf in the library to prevent the application from explicitly buffering its own stdout. And you'll want to wrap fwrite and printf so that they flush on each call. Writing the .so to be preloaded will take you outside of python.
Solution 3:
I don't think it's possible. If the source application doesn't flush its outgoing buffer, the data will not reach outside that process until the buffer overflows and a flush is forced.
Notice how a well-established command such as file has an option (-n) that causes it to flush its output explicitly. This is required when using file in the mode where it reads input file names from a pipe, and prints the detected type. Since in this mode, the file program doesn't quit when done, the output would otherwise not appear.
Consider this at a lower level: the output buffering simply means that doing write() on a buffered stream copies the data into an in-memory buffer, until the buffer fills up or (typically) until a linefeed is found. Then, the part of the buffer up to the overflow or linefeed is written write()n to the underlying system-level file descriptor (which could be a file, a pipe, a socket, ...).
I don't understand how you're going to convince that program to flush its buffer, from the outside.
Solution 4:
Its worth noting that some programs only buffer their output when they think it's not going to a "real user" (ie, a tty). When they detect that their output is being read by another program, they buffer.
The emulation of a tty is one of the things that Expect does in automating other processes.
There is a pure Python implementation of Expect, but I don't know how well it handles tty emulation.