Would you use Splunk?

Install the logcheck package. It will scan the logs once an hour and email you anything it doesn't consider normal. Essentially, it emails anything that entered the logs in the last hour that it doesn't have a rule for ignoring. There are additional attack rules than include things which shouldn't be in the log. The email subject line varies depending on the reason things were picked up.

I generally build a local ignore file for it as I discover things which I consider normal, but don't have existing ignore rules.

The various syslog alternatives all support server consolidation, so you can forward the logs to a single server. However, I haven't been in the habit of doing it. The only system I forward logs off of is my OpenWRT firewall.

EDIT: I do use Splunk at work to search log files, although if I known the particular log I am looking for I am more likely to use less. It does have alert capabilities, but we don't use them. I expect they would alert on a match to a known record. This can lead to a lot of false negatives if you have new problems without an alert rule. I prefer to have false positives like I get from logcheck. Splunk may have better timeliness on alerts though.

I do get timely alerts from fail2ban on cases that cause it to trigger. It also maintains blacklist entries for the originating source.


One other thing to add. Our company recently looked into purchasing Splunk. We definitely had more than 500MB of logs to analyze and we found that their licensing model was outrageously expensive. Splunk has taken advantage of their increase in popularity and slowly increased their prices over the years. When we first looked at it 2 years ago, the limit on free was 1GB and the licensing fees were half of what they are now.

Splunk is a fantastic tool, but at it's current price, I would think hard about alternatives IMHO.