How can I combine two commands to tally data from compressed and uncompressed log files?

Solution 1:

Can use zgrep which will decompress if necessary and therefore works with both straight text and compressed input. As well grep/zgrep can process multiple files directly which is needed in this case since mixed compressed and text to standard input doesn't always work as intended. Supress filenames in the grep output with -h or --no-filename.

sudo zgrep -h 'Failed password' /var/log/auth.* | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

Man page:

ZGREP(1)                                                              ZGREP(1)

NAME
       zgrep - search possibly compressed files for a regular expression

SYNOPSIS
       zgrep [ grep_options ] [ -e ] pattern filename...

DESCRIPTION
       Zgrep  invokes grep on compressed or gzipped files.  All options specified
       are passed directly to grep.  If no file is specified, then  the  standard
       input  is  decompressed if necessary and fed to grep.  Otherwise the given
       files are uncompressed if necessary and fed to grep.

       If the GREP environment variable is set, zgrep uses it as the grep program
       to be invoked.

AUTHOR
       Charles Levert ([email protected])

SEE ALSO
       grep(1), gzexe(1), gzip(1), zdiff(1), zforce(1), zmore(1), znew(1)

Solution 2:

While ugly to type each time, you could do something like:

for log in /var/log/auth.log*; do if ! sudo zcat "$log" 2>/dev/null; then sudo cat "$log"; fi; done | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

Probably it would be better to create a function for at least the cat part, something like:

getLogs() {
    for log in /var/log/auth.log*; do
        if ! sudo zcat "$log" 2>&/dev/null; then
            sudo cat "$log"
        fi
    done
}

Then your command can be

getLogs | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

or you can make the whole thing be a function:

getSSHFailures() {
    for log in /var/log/auth.log*; do
        if ! sudo zcat "$log" 2>/dev/null; then
            sudo cat "$log"
        fi
    done | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c
}

Then you just call

getSSHFailures

You could then define that function in your .bashrc and have that function available in each shell you start

Also, this should also fix both your observation about the error trying to zcat non compressed files (because we redirect zcat errors to /dev/null) and get rid of the duplicate entry, because we're printing all the results together as a single stream before processing them.

Finally, awk can actually do the pattern matching for us, so we can also get rid of the 2 greps like:

getSSHFailures() {
    for log in /var/log/auth.log*; do
        if ! sudo zcat "$log" 2>/dev/null; then
            sudo cat "$log"
        fi
    done | awk '/sshd.*Failed password/ {print $1,$2}' | sort -k 1,1M -k 2n | uniq -c
}