Does Prometheus supports webhooks?

We got a task to receive alerts from Prometheus to our internal system. The prometheus itself isn't alert manager, but just the Prometheus server, can we receive webhooks like the alert-manager?

Solution 1:

can we receive webhooks like the alert-manager?

You can receive HTTP POST requests from Prometheus if you configure your application as an Alertmanager:

# prometheus.yml
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - your-app.com:80

Though, this basically means that you have to reinvent the Alertmanager API, which isn't a smart move in my opinion because you will have to keep up with the API changes. It will be easier to make and maintain if you just run an Alertmanager instance and configure a webhook endpoint.

If the previous statement did not persuade you against the idea of receiving pushes directly from Prometheus, here are few things to help you solve the problem:

1. Sample POST request from Prometheus (v2.29.2):

[
    {
        "endsAt": "2022-01-20T18:03:41.955Z",
        "generatorURL": "http://prometheus.example.com:9090/graph?g0.expr=up+%3D%3D+1&g0.tab=1",
        "labels": {
            "alertname": "Test",
            "foo": "bar",
            "job": "test"
        },
        "startsAt": "2022-01-20T17:24:41.955Z"
    },
    ...
]

2. Alertmanager source code: https://github.com/prometheus/alertmanager
3. A simple HTTP server that prints out incoming requests. I used it to get p.1:

#!/usr/bin/env python3
import json
from flask import Flask, request


app = Flask(__name__)


@app.route('/', defaults={"err": ""})
@app.errorhandler(404)
@app.errorhandler(405)
def print_request(err):
    if request.json:
        body = json.dumps(request.json, indent=4, sort_keys=True)
    elif request.form:
        body = request.form
    else:
        body = request.data

    print(f"{request.method} {request.path}\n"
          "#################### HEADERS ###################\n"
          f"{request.headers}"
          "--------------------- BODY ---------------------\n"
          f"{body}")
    return "OK\n"


if __name__ == '__main__':
    app.run("0.0.0.0", port=8000)