How do I test Alertmanager alert groups about failed deploys before a real incident?

Tares lets you test Alertmanager alert groups about failed deploys by posting a sample batch to the ingest URL of an Alertmanager source. You then check the label mapping under Fields and the resulting events under Recent events on the source page. Nothing needs to fail in production, and the test batch has the same shape Alertmanager sends. When the events and labels look right, you point your real receiver at the same URL.

How it works

Alertmanager evaluates, groups and dedupes alerts, then POSTs a batch to your receiver. The Alertmanager connector is a push receiver. It takes that batch and fans out the alerts array into one event per alert:

• event_type is the alert name, for example a deploy failure alert.
• text is the alert’s summary annotation.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including its fingerprint, stays in the payload.

The entity key comes from a label automatically, in the order namespace, service, job, instance, or from a label you mark primary. The advanced key_label field lets you name a different alert label to key on.

Because the source only records what Alertmanager pushes, a hand-made batch behaves like a real one. A group of three failed deploys becomes three events, so you can test grouping by putting several alerts in one batch.

The test_source tool is not the way to check mapping on a push source. It takes no payload and only returns the ingest endpoint. To check the mapping, post a sample payload and read Recent events and Fields. This is the same method the webhook connector uses.

Set it up

1. Create the source. Open Sources, then Add source, then Alertmanager alerts, name it and click Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Copy the ingest URL from the source page. On a local Tares started with tares up, it starts with http://localhost:8787.

2. Send a sample failed deploy batch. Replace the ingest key with the one from your source page:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"DeployFailed","severity":"critical","namespace":"prod"},"annotations":{"summary":"Deploy of checkout failed"},"startsAt":"2024-01-01T00:00:00Z"}]}'

3. Check Recent events. You should see one event whose type is DeployFailed, whose text is the summary, and whose status label is firing.

4. Check Fields. It lists the alert labels that arrived and how often. Promote namespace or service to a label so Tares correlates failed deploys with your other sources. If you want a different key, set key_label.

5. Test a group and a resolution. Add a second object to the alerts array to see two events from one batch. Send the same alert again with status set to resolved to confirm the firing to resolved transition shows up on the timeline.

6. Point the real receiver at the same URL in alertmanager.yml, then reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

Keep send_resolved set to true so resolutions arrive. If your instance runs with tares up --auth, the sender needs a Bearer Authorization header holding an ingest-scoped API key. The source page shows the receiver block with the auth settings already filled in. Without auth on, the header is not needed.

What you get

• Proof that your failed deploy alerts arrive before an incident, as events with the right type, text and labels.
• A confirmed label mapping. Fields shows exactly which alert labels reach Tares, so you pick a reliable key such as namespace or service with real data in front of you.
• A verified firing and resolved pair for the same alert, so you know resolutions are tracked.
• A receiver block you can reuse as is for production. Only the sample payload changes.

From here, failed deploy events sit next to the rest of your sources, which is what the AI SRE workflow builds on.

When something else fits

• If the sender is not Alertmanager, such as a CI system or your own service, use the webhook connector. It accepts arbitrary JSON and lets you declare your own labels and event type field.
• If you only need to read alerts that Prometheus rules have fired and do not need Alertmanager’s grouping and silencing, the prometheus_alerts connector polls them instead.

Related