Log monitoring toolsets for Unix? [closed]
What are some good tools for monitoring logs on a system? I am currently using logsentry, which basically just runs grep on logs, ignores regexps I tell it to ignore, and mails me everything else, but I'd like to find something a little 'smarter', (ie can understand what a Postfix log is beyond just being a series of lines to grep on, things like that - contextual knowledge).
On services, not running anything particularly exotic - apache, memcached, postgres, postfix, openssh, squid, bind, mailman. About the strangest thing running is monotone, which I would not expect any log analyzer supports (thus, it would be nice if it is easy to extend with understanding of additional services).
Something that can parse and summarize information in hourly or daily reports (eg, 15 404s reported in this apache log file, 3 logins by this user from IPs x, y, and z, 5 mails sent by this user and 150 received by that other user, disk X is 90% full, 3000 failed ssh attempts from these random IPs, etc) would be ideal.
I've looked through several lists of log monitoring tools for Unix but most of them seem to boil down to tail -f or grep. I'm hoping there is something out there that I've overlooked.
Solution 1:
I think of monitoring in two types. There is monitoring the current state and keeping a history of that on the monitoring server, and also there is monitoring history (logs).
Some of the information you listed isn't really meant to be monitored via logs. Disk space for instance, would probably be better monitored by a plug-in that calls the df command or something like that. So I would consider this monitoring the current state, even if you keep a record of it.
I like Splunk for log monitoring, but it is expensive if you need it to do things the free version doesn't. For monitoring things like disk space, if a process is running, cpu usage, etc, personally I like Nagios. The two pair well together I think.
You might find something that does both of these well, but I wouldn't try to get one type to do the other (ie Nagios to monitor logs), it will probably be a little hackish.
Solution 2:
You can use Octopussy for all the reporting stuff you asked. (except the '90% disk full' as Kyle said probably better monitored by System monitors like Nagios, Zabbix,...)
Octopussy is a log management solution to:
- Search in your logs
- Raise Alerts from logs
- Generate Reports from logs
With Octopussy, you can 'easily' create reports on anything you want from any logs. (may be not so 'easy' because you need some work to define logs patterns & reports, but really powerfull after that first work)
Disclaimer: I work on Octopussy.
Solution 3:
What you want is OSSEC, seriously. It puts logsentry, logwatch and similars down easily.
It does exactly that by default, detecting web scans, brute forces and so many issues using log analysis (or log-based intrusion detection ,like they call it).
Try it out, it is the most easy tool to install (just run their install.sh script) and enjoy.
Link: http://www.ossec.net