Low-cost, Flexible Log Aggregation [closed]

Many applications can log to syslog, which means you can get the logs to a log server. apache, mysql, tomcat (log4j) can, at least.

Then you need a competent syslog server to do the aggregation. I use syslog-ng, but that's because it was the only serious alternative 7 years ago. Debian Lenny switched to rsyslog, which probably has a saner codebase and even more features.

In my experience, a good regex engine is the most important part of an aggregating syslog server. There is so much gorp you want to filter out so you can see the relevant parts. You can also point logwatch at the aggregated logs if you want to get started quickly.

EDIT: I should be explicit. Our strategy is to log everything from a specific host to one or more files in a folder for that host and simultaneously log to heavily filtered files that logs certain activities across all hosts. For example, there may be a file with failed logins across all hosts.

Another interesting project is Octopussy. It's an opensource log analyzer, alerter, and reporter. I haven't had a chance to set this up yet but I've heard some good things about it.

Someone else mentioned Zenoss. I use Zenoss and while it can alert based on logs I wouldn't say that's is it's strong point. It's primary task is monitoring and alerting based on snmp which is what I use it for.

LogZilla is highly scalable (hundreds of millions of events) and is 1/10 of the cost of the other software in it's class. It's also much easier to use.