Using AlertManager Webhooks

The defacto Alerting tool used with Prometheus is Alertmanager. Prometheus sends a notification to Alertmanager whenever a rule is triggered. Alertmanager then takes that notification and sends it to whatever receiver(s) you have defined in your config. It supports quite a few options: email, hipchat, pagerduty, pushover, slack, opsgenie, victorops, webhook, and wechat. For this post I’m focusing on the webhook receiver.

First let’s set up our Alert struct:

type Alert struct {
        Receiver string `json:"receiver"`
        Status   string `json:"status"`
        Alerts   []struct {
                Status string `json:"status"`
                Labels struct {
                        Alertname string `json:"alertname"`
                        Service   string `json:"service"`
                        Severity  string `json:"severity"`
                } `json:"labels"`
                Annotations struct {
                        Summary string `json:"summary"`
                } `json:"annotations"`
                StartsAt     string    `json:"startsAt"`
                EndsAt       time.Time `json:"endsAt"`
                GeneratorURL string    `json:"generatorURL"`
                Fingerprint  string    `json:"fingerprint"`
        } `json:"alerts"`
        GroupLabels struct {
                Alertname string `json:"alertname"`
        } `json:"groupLabels"`
        CommonLabels struct {
                Alertname string `json:"alertname"`
                Service   string `json:"service"`
                Severity  string `json:"severity"`
        } `json:"commonLabels"`
        CommonAnnotations struct {
                Summary string `json:"summary"`
        } `json:"commonAnnotations"`
        ExternalURL string `json:"externalURL"`
        Version     string `json:"version"`
        GroupKey    string `json:"groupKey"`
}

This is the payload that Alertmanager sends to your webhook. As you can see, there can be multiple alerts inside of this single message so be aware of that.

Now we just need a server to accept this payload. Let’s create a small webserver:

package main

import (
        "net/http"
        "log"
)

func alertFunc(w http.ResponseWriter, r *http.Request) {

}

func main() {
        http.HandleFunc("/", alertFunc)
        log.Fatal(http.ListenAndServe(":5001", nil))
}

With this outline set up, we just need to add our logic into our function.

First let’s initialize our struct and unmarshal our JSON.

defer r.Body.Close()

var alert Alert

body, err := ioutil.ReadAll(r.Body)
if err != nil {
        fmt.Println(err)
}

err = json.Unmarshal(body, &alert)
if err != nil {
        fmt.Println(err)
}

Now that we have our data from the payload unmarshaled into our struct, we can add some logic to act on the information.

if alert.Status == "firing" {
        for _, v := range alert.Alerts {
                if v.Labels.Severity == "critical" {
                        fmt.Println("The service " + v.Labels.Service + " is broken.")
                }
        }
}

Now clearly, this isn’t a really helpful thing we’ve done here. However, it’s just showing that based on the data we receive, we can act on specific parts of our payload. You will get a payload when the situation is resolved, so checking on firing makes sure that we are only doing something if there is an issue. You can also act differently based on the severity labels that are sent.

Here’s our final code (make sure to include the struct in another file):

package main

import (
        "encoding/json"
        "fmt"
        "io/ioutil"
        "log"
        "net/http"
)

func alertFunc(w http.ResponseWriter, r *http.Request) {
        defer r.Body.Close()
    
        var alert Alert

        body, err := ioutil.ReadAll(r.Body)
        if err != nil {
                fmt.Println(err)
        }

        err = json.Unmarshal(body, &alert)

        fmt.Println(alert.Status)

        if alert.Status == "firing" {
                for _, v := range alert.Alerts {
                        if v.Labels.Severity == "warning" {
                                fmt.Println("The service " + v.Labels.Service + " is broken")
                        }
                }
        }

}

func main() {
        http.HandleFunc("/", alertFunc)
        log.Fatal(http.ListenAndServe(":5001", nil))
}

One idea I had was to be able to self heal infrastructure with something like Terraform when specific labels are sent in the payload.

One thing to note is there is a library for alertmanager but I have not looked much into it. I do know it has a template for the data structures, but it was easy enough to use the bare JSON payloads here.