How do I test Alertmanager alert groups about a disk filling up before a real incident?

Tares lets you test a disk-filling alert before a real incident by posting a sample Alertmanager batch to the source’s ingest URL. You then read Recent events and Fields on the source page to confirm the event and the labels arrived as you expect. The test_source tool is not the check for this: on a push source it only returns the ingest endpoint.

How it works

The alertmanager connector is a push receiver. Alertmanager evaluates, groups and dedupes alerts, then POSTs a batch to the source’s ingest endpoint. Tares fans the alerts array out into one event per alert:

• event_type is the alert name.
• text is the alert’s summary.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including fingerprint, is kept in the payload.

The entity key comes from a label automatically, in the order namespace, service, job, instance, or from a label you declare primary. Set key_label if you want to key on a different alert label.

Because the connector only records what Alertmanager pushes, a hand-posted batch that has the same shape as a real one goes through the same path. That makes a sample payload a faithful rehearsal of the label mapping. The test_source tool does not replace it. For a push source it takes no payload and only returns the ingest endpoint, so it cannot check the label mapping.

For how arbitrary JSON is mapped when you post it directly, see the webhook connector. The overall picture of how alerts feed investigations is in AI SRE.

Set it up

1. Create the source. Ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The agent calls create_source and returns the ingest URL. In the console, go to Sources, then Add source, then Alertmanager alerts, name it and click Create. The source page shows a copy-paste receiver block with the ingest URL filled in.

2. Make sure Tares is running. On a local setup, start it with:

   tares up

3. Post a sample disk-filling alert to the ingest URL. The payload below has the same shape as the one on the Alertmanager connector page, with the alert name and summary changed to describe a disk:

   curl -X POST http://localhost:8787/ingest/<ingest_key> \
     -H 'Content-Type: application/json' \
     -d '{"alerts":[{"status":"firing","labels":{"alertname":"DiskFillingUp","severity":"critical","namespace":"prod"},"annotations":{"summary":"Disk is filling up"},"startsAt":"2024-01-01T00:00:00Z"}]}'

   If your instance is secured with tares up --auth, the sender needs a Bearer Authorization header holding an ingest-scoped API key. Without auth, the call works as written.

4. Open the source page and check Recent events. You should see one event whose type is DiskFillingUp, whose text is the summary, and whose labels include namespace, severity and status.

5. Open Fields. It lists the alert labels that arrived, so you can promote namespace or service to correlate on. Confirm the label you want as the entity key shows up with the value you expect.

6. When the sample looks right, add the receiver to your alertmanager.yml and reload Alertmanager:

   receivers:
     - name: tares
       webhook_configs:
         - url: http://localhost:8787/ingest/<ingest_key>
           send_resolved: true
   route:
     receiver: tares

   On a secured instance, the webhook_configs entry also carries the ingest-scoped API key as a Bearer Authorization header. The source page shows the auth block when the instance requires a token.

Repeat step 3 with different labels, for example another namespace, to see how the entity key changes before you rely on it.

What you get

• A rehearsal of the disk alert path that matches what Alertmanager will send, so you see the real event shape before an incident.
• A confirmed label mapping. Fields shows exactly which alert labels arrive, so you choose the entity key (namespace, service or a primary label) deliberately.
• A visible event in Recent events, with the alert name as the event type, the summary as the text and the synthetic status label.
• The full alert kept in the payload, fingerprint included, so nothing from the sample is lost.
• Confidence that the ingest URL and the receiver block are right before you reload Alertmanager.

When something else fits

• If you want Prometheus rule state polled on an interval instead of pushed in real time, use prometheus_alerts. Choose alertmanager when you want Alertmanager’s grouping, silencing and resolved handling.
• If the sender is not Alertmanager, or the payload has its own shape, the webhook connector takes arbitrary JSON and lets you declare labels for the fields you need.

Related