What should Alertmanager alert groups about failed deploys include for an AI SRE?

Tares turns every alert Alertmanager routes into one event, so each failed-deploy alert should carry a service label, a severity label, a summary annotation and a start time. Tares uses the alert name as the event type and the summary as the event text, and it keeps the full alert in the payload. Key the alert on service, and the agent can tie it to the deploy and the logs for that service.

How it works

Alertmanager has already evaluated, grouped and deduped the alerts. When it POSTs a batch to your Tares ingest URL, Tares fans the alerts array out into one event per alert:

• event_type is the alert name, so name the alert for what failed, for example a deploy failure, and not for a generic symptom.
• text is the alert’s summary annotation, so write a summary that says what broke in one line.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including fingerprint and the start time, stays in the payload.

The entity key, which decides which timeline the event lands on, comes from a label. Tares looks at namespace, then service, then job, then instance, or at a primary label you declare. This order is why the service label matters. An alert that carries both namespace and service keys on namespace by default. To key on service, set the advanced key_label field on the source to the service label, or declare service as the primary label. Then the alert sits on the same per-service timeline as your other sources, as described in Build an AI SRE.

The fields worth sending:

• Service name. A service label on every failed-deploy alert. It is the join key to metrics, logs and deploy events.
• Severity. A severity label such as critical. It arrives as a Tares label on the event.
• Summary. A summary annotation that names the service, the version or commit and the failure. It becomes the event text the agent reads first.
• Start time. startsAt on the alert. It is kept in the payload, so the agent can compare it with the deploy time and the first error.
• Resolved state. Set send_resolved: true so the resolution arrives as its own event with status set to resolved.

Set it up

1. Create the source. In the console, go to Sources → Add source → Alertmanager alerts, name it and click Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The source page shows a receiver block with your ingest URL filled in.

2. Add the receiver to your alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If Tares runs with auth on (tares up --auth), the sender uses a Bearer Authorization header holding an ingest-scoped API key. Without auth, leave it out.

4. In your alerting rule for failed deploys, set the alert name to the failure, add service and severity as labels, and add a summary annotation. If your alerts also carry a namespace label, set key_label to the service label in the source’s advanced settings so the alert keys on the service.

5. Send a test batch and check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

6. Open Fields on the source. It lists the alert labels, so you can promote service to correlate on.

7. Add the deploy itself as evidence. Create a webhook source with a primary service label read from the payload and event_type set to deploy, and have your CD pipeline POST to its ingest URL on every deploy. When the failed-deploy alert is keyed on service too, the alert and the deploy event share that service’s timeline. See the Alertmanager connector page for the receiver details.

What you get

• One event per alert, with the alert name as the event type and your summary as the text, so a failed deploy reads as a failed deploy.
• Labels the agent can use directly: the service, the severity and the firing or resolved status.
• The complete alert, with its fingerprint and start time, kept in the payload for the agent to check against deploy and log times.
• A shared timeline with your other service-keyed sources when the alert keys on service. The agent can read the alert next to the deploy event and the error logs without extra lookups.
• Resolved events when send_resolved is on, so the timeline shows when the failure ended.

Alertmanager keeps owning grouping, silencing and routing. Tares records what it pushes and correlates it with your other sources.

When something else fits

• If you do not run Alertmanager and only have Prometheus rules, the prometheus_alerts connector polls fired alerts instead. It does not give you Alertmanager’s grouping, silencing or real-time delivery.
• If the failed deploy never raises an alert, send it straight from your CD pipeline to a webhook source as a deploy event, and skip the alert path for that signal.

Related