Why does tailing logs with kubectl prevent pipe from producing output? [duplicate]

I've gotten into an interesting state on Ubuntu. The steps below describe it the best.

With a single pipe, I'm seeing what I expect

# In shell A
tail -f foo.log | grep aaa

# In shell B
echo aaa >> foo.log

# Shell A prints out `aaa`

But with multiple pipes I'm not seeing anything at all

# In shell A
tail -f foo.log | grep aaa | grep bbb

# In shell B
echo aaa bbb >> foo.log

# Nothing ever prints in shell A

But it works fine if I'm just echoing.

echo 'aaa bbb' | grep aaa | grep bbb

This is my attempt to create a minimal reproduction -- I originally encountered the issue trying to tee logs from adb logcat (Android dev tools). I've tried in zsh, bash, and fish as well.

I assumed it has something to do with my inotify watcher limit but bumping it didn't change anything.

This is because of buffering in the pipe, which in general doesn't care about lines and can accumulate data.

I think tail -f uses line buffering by itself; and the last grep writes to the tty, so it also uses line buffering. Therefore your first example works.

But grep in the middle is different and you need to adjust its behavior by forcing line buffering or by disabling buffering. The bellow commands will work as you expected.

• If your grep supports --line-buffered (it does in Ubuntu):

    tail -f foo.log | grep --line-buffered aaa | grep bbb
  

• More generic solutions (they will work with many filters other than grep):

    tail -f foo.log | unbuffer -p grep aaa | grep bbb
    tail -f foo.log | stdbuf -oL grep aaa | grep bbb
    tail -f foo.log | stdbuf -o0 grep aaa | grep bbb
  

See man 1 grep, man 1 unbuffer and man 1 stdbuf for details and quirks.

Notes:

• Neither solution is portable (grep --line-buffered, unbuffer and stdbuf are not specified by POSIX).
• If you can do this with grep --line-buffered then it should be your choice. There's no point in using extra tools.
• Related question on Unix & Linux SE: Turn off buffering in pipe.
• unbuffer and stdbuf work in completely different ways.
• With stdbuf, line buffering (-oL) should be preferred over no buffering (-o0) here, because
• it most likely performs better,
• and other parts of your pipe use line buffering anyway.
• If your last grep wrote to yet another file, it would behave like the other grep. In such case if you want lines to appear in the final file immediately, then you should also modify the behavior of the last grep.
• In fish, if grep is a wrapper function, you may not get the desired behavior with --line-buffered. Use command grep --line-buffered. See this question: Output pipe waits for EOF in fish.

Side note: tail foo.log | grep aaa | grep bbb (i.e. tail without -f) does not cause the issue because tail exits. When tail exits, the first grep detects EOF, flushes its buffer and exits, then the second grep does the same.