Multiple pipes on a tailed log - last pipe never receives stdin
This is because of buffering in the pipe, which in general doesn't care about lines and can accumulate data.
I think tail -f
uses line buffering by itself; and the last grep
writes to the tty, so it also uses line buffering. Therefore your first example works.
But grep
in the middle is different and you need to adjust its behavior by forcing line buffering or by disabling buffering. The bellow commands will work as you expected.
-
If your
grep
supports--line-buffered
(it does in Ubuntu):tail -f foo.log | grep --line-buffered aaa | grep bbb
-
More generic solutions (they will work with many filters other than
grep
):tail -f foo.log | unbuffer -p grep aaa | grep bbb tail -f foo.log | stdbuf -oL grep aaa | grep bbb tail -f foo.log | stdbuf -o0 grep aaa | grep bbb
See man 1 grep
, man 1 unbuffer
and man 1 stdbuf
for details and quirks.
Notes:
- Neither solution is portable (
grep --line-buffered
,unbuffer
andstdbuf
are not specified by POSIX). - If you can do this with
grep --line-buffered
then it should be your choice. There's no point in using extra tools. - Related question on Unix & Linux SE: Turn off buffering in pipe.
-
unbuffer
andstdbuf
work in completely different ways. - With
stdbuf
, line buffering (-oL
) should be preferred over no buffering (-o0
) here, because - it most likely performs better,
- and other parts of your pipe use line buffering anyway.
- If your last
grep
wrote to yet another file, it would behave like the othergrep
. In such case if you want lines to appear in the final file immediately, then you should also modify the behavior of the lastgrep
. - In
fish
, ifgrep
is a wrapper function, you may not get the desired behavior with--line-buffered
. Usecommand grep --line-buffered
. See this question: Output pipe waits for EOF infish
.
Side note: tail foo.log | grep aaa | grep bbb
(i.e. tail
without -f
) does not cause the issue because tail
exits. When tail
exits, the first grep
detects EOF, flushes its buffer and exits, then the second grep
does the same.