Using AlertManager Webhooks
The defacto Alerting tool used with Prometheus is Alertmanager. Prometheus sends a notification to Alertmanager whenever a rule is triggered. Alertmanager then takes that notification and sends it to whatever receiver(s) you have defined in your config. It supports quite a few options: email, hipchat, pagerduty, pushover, slack, opsgenie, victorops, webhook, and wechat. For this post I’m focusing on the webhook receiver. First let’s set up our Alert struct: type Alert struct { Receiver string `json:"receiver"` Status string `json:"status"` Alerts []struct { Status string `json:"status"` Labels struct { Alertname string `json:"alertname"` Service string `json:"service"` Severity string `json:"severity"` } `json:"labels"` Annotations struct { Summary string `json:"summary"` } `json:"annotations"` StartsAt string `json:"startsAt"` EndsAt time....