Sensu only handling every 180 occurrences

I have the latest sensu-core installed and I've installed the sensu-plugin gem using the omnibus package.

Sensu is up and running no problem, but I'm running into major issues with handlers. I've setup both the standard ruby mailer.rb from the community plugins, as well as the pagerduty handler.

When I add the handlers to a check that I know is failing in the logs I see this:

{\"id\":\"a1e807d2-425f-4891-b291-ed5bda5b6105\",\"client\":{\"name\":\"mfp\",\"address\":\"25.147.26.157\",\"subscriptions\":[\"test\"],\"version\":\"0.20.0\",\"timestamp\":1437673191},\"check\":{\"command\":\"/etc/sensu/plugins/check-disk.sh -d /dev/md1 -w 80 -c 90\",\"interval\":10,\"handlers\":[\"default\",\"pagerduty\",\"mailer\"],\"occurrences\":3,\"subscribers\":[\"test\"],\"name\":\"disk\",\"issued\":1437673279,\"executed\":1437673201,\"duration\":0.01,\"output\":\"CRITICAL - /dev/md1 space used=94% | '/dev/md1 usage'=94%;80;90;\\n\",\"status\":2,\"history\":[\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\",\"2\"],\"total_state_change\":0},\"occurrences\":513,\"action\":\"create\"}"]}
{"timestamp":"2015-07-23T12:41:20.873279-0500","level":"info","message":"handler output","handler":{"type":"pipe","command":"/etc/sensu/handlers/pagerduty.rb","api_key":"2e24c101ef754ff6add39aaaffa846c9","name":"pagerduty"},"output":["only handling every 180 occurrences: ws/disk\n"]}
{"timestamp":"2015-07-23T12:41:20.873561-0500","level":"info","message":"handler output","handler":{"type":"pipe","command":"/etc/sensu/handlers/pagerduty.rb","api_key":"2e24c101ef754ff6add39aaaffa846c9","name":"pagerduty"},"output":["only handling every 180 occurrences: wwv/disk\n"]}
{"timestamp":"2015-07-23T12:41:21.055407-0500","level":"info","message":"handler output","handler":{"type":"pipe","command":"/etc/sensu/handlers/pagerduty.rb","api_key":"2e24c101ef754ff6add39aaaffa846c9","name":"pagerduty"},"output":["only handling every 180 occurrences: mfp/disk\n"]}

So what it looks like is the handler is firing but being filtered by the sensu-handler.rb gem and not creating emails or pagerduty events.

Here's the offending line

only handling every 180 occurrences

Does anyone know what the problem is? Is this a bug? Do I need to override something? I have a ton of occurrences in sensu for this particular check (check_disk) so it should be firing to pagerduty and the mailer.

I faced the same issue trying to integrate a slack handler. You have to add the refresh parameter in your handler config file. For example :

{
  "checks": {
    "memory": {
      "command": "/etc/sensu/plugins/check-memory.sh -w 700 -c 64",
      "interval": 10,
      "subscribers": [
        "test"
      ],
      "handlers": [
        "slack",
        "default"
      ],
      "occurrences": 5,
      "refresh": 60
    }
  }
}

Solution found in this tutorial: http://www.whiteboardcoder.com/2014/11/getting-sensu-to-talk-to-hipchat.html

This is a bug in sensu-plugin, well not necessarily a bug but a lack of documentation. The problem is in config.json you need to specify the host declarative for the API block, sensu-plugin needs this otherwise you run into these filtering issues.

  "api": {
    "host": "localhost",
    "port": 4567
  }

Sensu only handling every 180 occurrences

Related

Recent Posts