Logstash, Kibana and email alerts

I am trying to setup email alerts using logstash. Right now it emails me EVERY time the pattern "Error" is parsed into my log file which can lead to a lot of unnecessary emails. I'd like to create a conditional rule so that let's say "X logfile has the pattern Error 3x in 1 minute email me". This way I don't get overwhelmed with emails.

Here is my current config:

input {
  file {
#    sincedb_path => /path/to/whatever/
    path => "/opt/test.log"
    type => "test_log"
  }
}

filter {
   dns {
      add_field => [ "IPs", "Logs, from %{host}" ]
      type => [ "MESSAGES" ]
      resolve => [ "host" ]
      action => [ "append" ]
     }
}

filter {
  if [message] == "Error" or [message] == "error" {
    throttle {
      before_count => 1
      after_count => 3
      period => 10
      key => "%{message}"
      add_tag => "throttled"
  }
} }

output {
#  stdout { codec => rubydebug }
   redis { host => "redis_IP" data_type => "list" key => "logstash" }
   if "throttled" not in [tags] {
      email {
        from => "[email protected]"
        to => "[email protected]"
        subject => "Alert from  %{path}, from %{host}"
        body => "Message is: ]\n'%{message}'. \nLog file:\n %{path}:\n\n%{message}.\n More information can be viewed in Kibana"
        }
    }
}

We just setup Riemann to handle alerting based on log messages.

Riemann can read a stream of log messages from logstash and send out alerts based on the contents.

One of the advantages with riemann is you can rollup all messages from a certain time into one email. This way you will not get to many e-mails but you will still get all your messages.

Much more examples can be found at http://riemann.io/howto.html