How to apply a filter to real time output of `tail -f `?

tail -f path

The above will output modifications to the file instantly, but I want to apply a filter to the output, only show when there is a keyword xxx in it.

How to approach this?

Solution 1:

With Unix you can pipe the output of one program into another.

So to filter tail, you can use grep:

tail -f path | grep your-search-filter

Solution 2:

Short answer: tail -f somefile | grep somepattern

However, this tends to fall short. Let's say you're tailing a file that gets rotated often (if its a debug log, it might be rotated multiple times). In that case tail -F is your friend. I'll let you look up the difference.

But tail -f and tail -F print out a bunch of lines first, which is often undesirable in this use-case, so in this case add -n0

tail -F -n0 somefile | grep somepattern

That will be fine, up until you want to do some other filtering, and then you need to beware of buffering. stdout is line-buffered by default when writing to a terminal but when fully-buffered when writing to a pipe. So the following will emit lines as soon as they are found, because tail is explicitly line-buffered (or it flushes its output at the end of each line), and grep is also line-buffered because its output is going to your terminal:

tail -F -n0 somefile | grep somepattern

But then you decide to use something like awk or cut to further process the output.

tail -F -n0 somefile | grep somepattern | awk '{print $3}'

And now you wonder where your output has gone... depending on the volume of logs, you may find that you do get output, but it will be a page at a time because now the stdout of grep is operating in fully-buffered fashion, and so awk receives it input 4kB at a time (by default).

In this case, you can tell grep to always make stdout line buffered by using the --line-buffered option.

tail -F -n0 somefile | grep --line-buffered somepattern | ...

However, most commands don't have an analogue of --line-buffered. In the case of more scriptable tools, you can use a function to flush the output (eg. in awk, the function is fflush(), which shares the same name as its C counterpart, tools like Perl and Python have something similar).

With the likes of cut you're likely out of luck; ... but you might try searching for unbuffer, which is I think something provided by the expect toolchain (I've never used it).

I hope you've found this useful.

Cheers, Cameron