Using AlertManager Webhooks

The defacto Alerting tool used with Prometheus is Alertmanager. Prometheus sends a notification to Alertmanager whenever a rule is triggered. Alertmanager then takes that notification and sends it to whatever receiver(s) you have defined in your config. It supports quite a few options: email, hipchat, pagerduty, pushover, slack, opsgenie, victorops, webhook, and wechat. For this post I’m focusing on the webhook receiver. First let’s set up our Alert struct: type Alert struct { Receiver string `json:"receiver"` Status string `json:"status"` Alerts []struct { Status string `json:"status"` Labels struct { Alertname string `json:"alertname"` Service string `json:"service"` Severity string `json:"severity"` } `json:"labels"` Annotations struct { Summary string `json:"summary"` } `json:"annotations"` StartsAt string `json:"startsAt"` EndsAt time....

October 19, 2019 · 3 min · 555 words · John Hooks

Using Make for SysAdmin Tasks

When most people hear about make they think of compiling source code. While that is often what it’s used for, there are other good uses as well. Here’s an example of using it with Terraform/Ansible to build your infrastructure. Make has some advantages over just using a shell script. One is the native concept of dependencies. You can easily define which targets are dependencies of other targets. Make also has some idempotence baked in (as long as everything isn’t a phony target)....

August 11, 2019 · 3 min · 609 words · John Hooks

Ansible Modules with Go

Since Ansible 2.2 you can use binary applications as modules for Ansible. This means you can write modules in languages other than Python. The downside is that the modules aren’t integrated as well as if they were written in Python with Ansiballz. The binary modules only takes the filename as an argument which is a temporary file containing the JSON data of the modules parameters. I took the boilerplate code that Ansible had here and created a small module to generate a random password....

June 30, 2019 · 3 min · 499 words · John Hooks

Ansible and Jenkins Remote Triggers

Ansible Tower has an awesome feature that I’ve talked about before called provisioning callbacks. It’s hard to beat their simplicity, one click of a button and you have an API endpoint and a key to use. However you can replicate this in Jenkins with just about the same amount of work. Coming from using both Tower and Jenkins, I currently prefer to use Jenkins. I find it a little more flexible....

June 18, 2019 · 2 min · 312 words · John Hooks

Transient Playbooks for Ansible

So I had a unique scenario where we built a solution for users using Ansible but wanted to make it simple so that it didn’t require the end users to understand how to build an inventory or playbooks. So the solution we came up with was to build Ansible playbooks and inventories with Ansible itself. The users just need a single YAML file and then run a make command and Ansible builds itself the inventory, playbooks, and group_vars at run time....

June 17, 2019 · 3 min · 463 words · John Hooks

Goniq

In my journey to better my development skills I’ve been trying to write some small Go applications. I needed a sorter like uniq but on a Windows machine for a project I was doing. So I decided to write one. Unlike uniq the input does not need sorted beforehand. The program takes either stdin or can read from a file which is passed as the first argument. stdin: echo "test thing test thing other" | goniq...

May 25, 2019 · 1 min · 148 words · John Hooks

Nomad

Nomad is a job scheduler created by Hashicorp. Since it’s written in Go, it’s a single, cross platform, statically linked binary. It has drivers for Docker, Qemu, Rkt, Java, App execution, and a beta for LXC. I’ll just go over the exec driver in this post. To start you need to download Nomad from here. I stuck it in /usr/local/bin. I used Vagrant to bring up a server and two clients....

July 7, 2018 · 3 min · 515 words · John Hooks