piping in real time

This is more a conceptual thing than a current problem.

Would it be possible to do this:

tail -f <file> | grep "string" | sed 's/stuff//g' >> output.txt

The problem I'm thinking is that tail -f never terminates so it won't progress in the command. I'm trying to read and output a file in real-time. If this doesn't work how would I do it?

Solution 1:

Most commands that output data on stdout, (which includes grep and sed,) buffer their output when it's not going to a terminal. That is, they wait for a large chunk of output to gather (like 4KiB) before flushing the chunk to the file or pipe. This is generally more efficient. Some commands allow you to override this, like GNU grep which has a --line-buffered option, and GNU sed has an --unbuffered option. For commands that don't have a way to override the buffering, you can use tools like unbuffer or stdbuf.

In this case though, I'd consider using GNU awk instead, where you can control the flushing of stdout yourself (with the fflush function).

tail -f file | gawk '/string/{ gsub(/stuff/, "") } { print; fflush() }' > output.txt

See BashFAQ #9 for more on buffering.