Piping tail -f into awk

I am trying to pipe tail -f into awk to monitor a logfile in realtime, but according to examples, there should be not problem but I can't get it to work.

here is the command I'm running

tail -f logfile.log | awk -F" " '{print $1, $2, $7, $8}'

But when I edit my file using nano add a line, it is not printed in real time, If I run the awk command directly, my new line appear in the result.

You don't see it in real time because, for purposes of efficiency, pipes are buffered. tail -f has to fill up the buffer, typically 4 kB, before the output is passed to awk.

A fix is to use the unbuffer command which is part of the expect package:

unbuffer tail -f logfile.log | awk -F" " '{print $1, $2, $7, $8}'

This tricks tail into thinking it is writing to an interactive terminal. As a result, it doesn't buffer.

For more, see https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe

Alternatively, if you have GNU coreutils 7.5 or better, you can disable output buffering with the stdbuf command:

stdbuf -o0 tail -f logfile.log | awk -F" " '{print $1, $2, $7, $8}'

@John1024's alternative answer is not quite correct, as the stdbuf -o0 prefix is misplaced. It belongs as the prefix to the awk command, not the tail -f command, so the correct command should be:

tail -f logfile.log | stdbuf -o0 awk -F" " '{print $1, $2, $7, $8}'

This is what worked for me, BUT only after I located an awk version that would cooperate. So note that not all versions of awk allow this! I had to try nawk, gawk, mawk, etc until I found one that worked. If you need that particular command string to function as written, keep trying various awk/gawk/mawk versions by being explicit with the path of the awk binary and even downloading others until you hit one that works.

On Ubuntu 20.10 where awk links to mawk version 1.3.4,

Add -W interactive. E.g. tail -f $file | awk -W interactive '{print}'

(Thanks to the other folks whose answers here caused me to read the man page looking for buffer.)

You could un-buffer it by adding a while loop as given in the example below. I have tested this on a running log file and it worked for me.

tail -f input.log | while read a; do echo "$a" | awk -F" " '{print $1, $2, $7, $8}' >> output.log; done

Neither stdbuf nor unbuffer seem to work for me (Ubuntu 18.04), but using a named pipe does. I created a shell script containing:

mkfifo /tmp/pipe
awk -F" " '{print $1, $2, $7, $8}' < /tmp/pipe &
tail -f input.log > /tmp/pipe