How do I test Alertmanager alert groups about crash loops before a real incident?

Tares lets you test crash loop alerts before a real incident by posting a sample Alertmanager batch to your source’s ingest URL. You then check the event on the timeline and the labels under Fields. For a push source like alertmanager, test_source takes no payload and only returns the ingest endpoint, so the sample post is what checks your label mapping.

How it works

The alertmanager connector is a push receiver. Alertmanager POSTs a batch shaped like { alerts: [...] } to the source’s ingest endpoint, and Tares fans the alerts array out into one event per alert.

For each alert, Tares records:

• event_type: the alert name.
• text: the alert’s summary.
• Labels: the alert’s own labels, plus a synthetic status of firing or resolved.
• Payload: the full alert, including fingerprint.

The entity key is derived automatically from a label, checked in this order: namespace, service, job, instance, or a label you declare primary. If you want a different label as the key, set the advanced key_label field.

Because you send the same shape Alertmanager sends, a hand-made crash loop batch exercises the same path a real one takes. Details of the connector are on the Alertmanager connector page. The generic push path behind it is described on the webhook connector page.

Set it up

1. Create the source. Go to Sources → Add source → Alertmanager alerts, name it and select Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The agent calls create_source and returns the ingest URL.
2. Add the receiver to your alertmanager.yml and reload Alertmanager. The source page shows this block with the ingest URL already filled in.

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If your Tares has auth on (tares up --auth), the sender uses a Bearer Authorization header holding an ingest-scoped API key. Alertmanager takes it in the http_config of the webhook_configs entry, and the source page shows the block. Without auth on, you need no key.
4. Post a sample crash loop batch. Use the same shape as the documented test, with a crash loop alert name, summary and the labels you expect in production:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"KubePodCrashLooping","severity":"critical","namespace":"prod"},"annotations":{"summary":"Pod is crash looping"},"startsAt":"2024-01-01T00:00:00Z"}]}'

5. Open the source page and check Recent events. You should see one event whose type is the alert name and whose text is the summary.
6. Open Fields and check that the alert labels appear. Promote namespace or service there if you want to correlate on it.
7. Post the same alert again with "status":"resolved" to confirm the resolved state shows up in the synthetic status label.

If you also want to confirm Alertmanager itself routes to Tares, make sure your crash loop alerts match the route that points at the tares receiver, then wait for or trigger a real alert in a non-production environment.

What you get

• A confirmed path from your Alertmanager config to Tares, checked before anything is on fire.
• Proof of the label mapping: you see which alert labels arrive and how the entity key is derived.
• A look at the event on the timeline, with its type, text and status.
• The full alert, including fingerprint, kept in the payload so you can inspect what Alertmanager sent.
• A chance to pick the label you want as the key, using key_label or a primary label, before a real crash loop depends on it.

Tares records what Alertmanager pushes. Alertmanager has already evaluated, grouped and deduped the alerts, and with send_resolved it tracks firing to resolved. Your test therefore reflects the real delivery.

From here you can explore your alerting setup in the AI SRE guide.

When something else fits

• If you would rather have Tares read alerts from Prometheus on a schedule than receive them from Alertmanager, use prometheus_alerts. You give up Alertmanager’s grouping, silencing and real-time delivery.
• If your sender is not Alertmanager, use the webhook connector and post a sample payload the same way, then check Recent events and Fields.

Related