How can I monitor syslog and send notification based on events?

Solution 1:

^{Question seems to have changed a bit and targetted on scripting, rather than sending notifications. I might remove this answer later.}

^{It seems it is quite straightforward to set up real time triggering of shell scripts based on messages passed to rsyslog. If OP rewrites his answer to contain a better picture of what he is aiming for (as already posted in comments), I am willing to provide a high quality answer here.}

Not exactly sure what your requirements are, but one particular piece of software I could recommend is logcheck. This may be useful in your situation too.

Its purpose is to send mail digests on specific matched (interesting) lines. I'm assuming basic mail configuration is working (e.g. echo bla | mail -s testmail [email protected] works) already. It takes care of log rotations and it knows where it has left of, so don't worry about that in scripting anymore!

Extracted some of the essential steps from this tutorial:

1. Configure basic things in /etc/logcheck/logcheck.conf:

   REPORTLEVEL="server"          # default reportlevel with pre-configured filters
   SENDMAILTO="[email protected]"
   

2. Which log files to check in /etc/logcheck/logcheck.logfiles:

   /var/log/syslog
   /var/log/auth.log
   /var/log/myapp.log
   

3. Now, to tell it to trigger on a specific line similar to

   myapp: Error: Login failed for admin123
   

   for this myapp log, this can be realized by creating a regular expression in /etc/logcheck/ignore.d.server/myapp (note server corresponding to the used REPORTLEVEL earlier on):

   ^myapp:.*
   

   _{Okay, this is not the most specific regex, but that's not the point.}

4. Wait emails to arrive, e.g.:

   Subject: webserver.domain.tld 2013-01-01 10:17 System Events
   
   Jan 01 09:40:59 webserver myapp: IMAP Error: Login failed for gertvdijk