How can I combine two commands to tally data from compressed and uncompressed log files?
Solution 1:
Can use zgrep
which will decompress if necessary and therefore works with both straight text and compressed input. As well grep/zgrep can process multiple files directly which is needed in this case since mixed compressed and text to standard input doesn't always work as intended. Supress filenames in the grep output with -h
or --no-filename
.
sudo zgrep -h 'Failed password' /var/log/auth.* | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c
Man page:
ZGREP(1) ZGREP(1)
NAME
zgrep - search possibly compressed files for a regular expression
SYNOPSIS
zgrep [ grep_options ] [ -e ] pattern filename...
DESCRIPTION
Zgrep invokes grep on compressed or gzipped files. All options specified
are passed directly to grep. If no file is specified, then the standard
input is decompressed if necessary and fed to grep. Otherwise the given
files are uncompressed if necessary and fed to grep.
If the GREP environment variable is set, zgrep uses it as the grep program
to be invoked.
AUTHOR
Charles Levert ([email protected])
SEE ALSO
grep(1), gzexe(1), gzip(1), zdiff(1), zforce(1), zmore(1), znew(1)
Solution 2:
While ugly to type each time, you could do something like:
for log in /var/log/auth.log*; do if ! sudo zcat "$log" 2>/dev/null; then sudo cat "$log"; fi; done | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c
Probably it would be better to create a function for at least the cat
part, something like:
getLogs() {
for log in /var/log/auth.log*; do
if ! sudo zcat "$log" 2>&/dev/null; then
sudo cat "$log"
fi
done
}
Then your command can be
getLogs | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c
or you can make the whole thing be a function:
getSSHFailures() {
for log in /var/log/auth.log*; do
if ! sudo zcat "$log" 2>/dev/null; then
sudo cat "$log"
fi
done | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c
}
Then you just call
getSSHFailures
You could then define that function in your .bashrc
and have that function available in each shell you start
Also, this should also fix both your observation about the error trying to zcat
non compressed files (because we redirect zcat
errors to /dev/null
) and get rid of the duplicate entry, because we're printing all the results together as a single stream before processing them.
Finally, awk
can actually do the pattern matching for us, so we can also get rid of the 2 grep
s like:
getSSHFailures() {
for log in /var/log/auth.log*; do
if ! sudo zcat "$log" 2>/dev/null; then
sudo cat "$log"
fi
done | awk '/sshd.*Failed password/ {print $1,$2}' | sort -k 1,1M -k 2n | uniq -c
}