How do I wake an AI SRE once when Alertmanager alert groups report failed deploys?

Tares wakes your AI SRE once per entity when Alertmanager reports failed deploys: you point an Alertmanager receiver at an alertmanager source and add a trigger with a cooldown. Each alert becomes one event, and the trigger evaluates per entity. While the alert keeps repeating inside the cooldown, the same entity does not fire again. Resolved alerts are still recorded as events, so the timeline shows the full firing to resolved story.

How it works

• One event per alert. Alertmanager POSTs a batch with an alerts array. The alertmanager source in Tares splits it into one event per alert. The event_type is the alert name, the text is its summary, and the alert labels become Tares labels plus a synthetic status of firing or resolved. The full alert, including fingerprint, stays in the payload. See Alertmanager for the connector details.
• One entity per service. Tares derives the entity key from a label, in the order namespace, service, job, instance, or from a label you declare primary. You can also set key_label on the source.
• One firing per entity per cooldown. A trigger checks its condition as events arrive. With group_by: [key_value] it evaluates per entity, and cooldown is the minimum gap between firings for the same entity. A repeating failed-deploy alert for prod fires once, then stays quiet for that entity until the cooldown passes. A different entity fires on its own.
• Context with the wake-up. When the trigger fires, Tares POSTs a dispatch to each subscriber. The payload is the entity’s correlated timeline over context_window, so the agent starts with the context it needs and makes no follow-up call.
• Resolved events land on the timeline. With send_resolved: true, Alertmanager sends the resolved notification, and Tares stores it as an event with status set to resolved. It shows up in Recent events and in the correlated timeline next to your other sources.

Set it up

1. Create the source. Ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Or use Sources → Add source → Alertmanager alerts, name it and select Create. The source page shows a receiver block with the ingest URL filled in.
2. Add the receiver to alertmanager.yml and reload Alertmanager. Keep send_resolved: true so resolved alerts reach the timeline:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If your Tares instance has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. The source page shows the matching block. Without auth you need nothing more.
4. Send a test batch and check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

5. Open Fields on the source and promote namespace or service as the label to correlate on.
6. Add the trigger on the project’s page, under Triggers → Add trigger, or in the catalog YAML. Replace DeployFailed with the alert name your rule uses, and deploy-alerts with your source name:

triggers:
  - name: failed_deploy
    sources: [deploy-alerts]
    key_field: namespace
    filters:
      - { field: event_type, op: contains, value: DeployFailed }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: failed_deploy
      context_window: 15m
    cooldown: 30m

7. Subscribe the agent. A Tares agent is woken by the trigger once you wire it to the trigger in a project and turn it on. An external agent registers a webhook with the subscribe tool: “Use tares to subscribe to the failed_deploy trigger and deliver to https://my-agent.example.com/hook.”

The filter matches on event_type only, so firing and resolved alerts of the same name both stay in the timeline handed to the agent. Filters narrow which events are handed over, so a filter on status would drop the resolved events from the payload.

Choose the cooldown to match how long a failed deploy normally takes to be fixed or retried. Every firing is logged on the trigger’s page under Firings, even with zero subscribers, so you can check how often the trigger fires and tune the value.

What you get

• One investigation per entity for a repeating failed-deploy alert, instead of one per Alertmanager repeat.
• Independent firing per entity, so a failed deploy in prod does not hide one in staging.
• A dispatch whose payload is the entity’s correlated timeline: the failed-deploy alerts, the resolved notification when it arrives, and events from your other sources over the context window.
• A record of every firing and every delivery outcome in the console, plus a dispatch page with the exact payload sent.
• Delivery you can rely on: at-least-once, retried on 5xx and transport errors, so dedupe on dispatch_id in your endpoint.

When something else fits

• If you want Tares to poll Prometheus for fired alerts on a clock instead of receiving Alertmanager’s grouping and resolved handling, use the prometheus_alerts connector.
• If you want a periodic summary instead of a reaction to each failure, give the trigger a schedule with condition: {every: 10m}, as described in Watch with triggers.
• For the wider picture of how Tares investigates incidents, start at AI SRE.

Related