How do I wake an AI SRE once when Alertmanager alert groups report 5xx errors on an API?

Tares wakes your AI SRE once per entity when you send Alertmanager to an alertmanager source and put a trigger with a cooldown on it. Each alert in a group becomes one event, the trigger evaluates per entity, and the cooldown holds back repeats, so a repeating 5xx alert starts one investigation. Resolved alerts arrive as events with status resolved, so the end of the incident shows up next to the start.

How it works

• Alertmanager POSTs a batch of alerts to the source’s ingest URL. Tares fans the alerts array out into one event per alert. The event_type is the alert name, the text is its summary, the alert labels become Tares labels, and a synthetic status label is firing or resolved. The full alert, including fingerprint, stays in the payload.
• The entity key comes from a label, checked in the order namespace, service, job, instance, or from a label you declare as primary. You can also set key_label on the source to choose the label yourself.
• A trigger watches the source, checks its condition as events arrive, and with group_by: [key_value] evaluates it per entity. Each service or namespace fires independently.
• cooldown is the minimum gap between firings for the same entity. A flapping or repeating 5xx alert for one API therefore wakes the agent once per cooldown window, while a different API can still fire on its own.
• When the trigger fires, Tares POSTs a dispatch to each subscriber. Its payload is the entity’s correlated timeline over context_window, so the agent starts with the alert, logs and deploys around it. Each firing is logged under Firings, even with zero subscribers.

This is the alerting side of AI SRE: Alertmanager does the grouping and silencing, and Tares decides when to wake an agent.

Set it up

1. Create the source. Ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Or open Sources → Add source → Alertmanager alerts, name it and select Create. The source page shows a receiver block with the ingest URL filled in.

2. Add the receiver to alertmanager.yml and reload Alertmanager. Keep send_resolved: true so resolved alerts reach Tares.

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If Tares runs with auth on (tares up --auth), the Alertmanager sender uses a Bearer Authorization header holding an ingest-scoped API key. The source page shows the exact block to paste. Without auth, you need nothing extra.

4. Send a test batch and check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

5. Open Fields on the source and check which label names your API. If it is namespace or service, the key is already derived. Otherwise declare a primary label or set key_label.

6. Add the trigger on the project’s page, under Triggers → Add trigger, or in the catalog YAML. Replace ApiHigh5xxRate with the alert name from your Alertmanager rule:

triggers:
  - name: api_5xx_alert
    sources: [api_alerts]
    key_field: service
    filters:
      - { field: event_type, op: eq, value: ApiHigh5xxRate }
      - { field: status, op: eq, value: firing }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: api_5xx
      context_window: 15m
    cooldown: 30m

7. Wake an agent. A Tares agent is woken by the trigger in a project that wires it to that trigger, once you turn it on there. An external agent registers a webhook with the subscribe tool: “Use tares to subscribe to the api_5xx_alert trigger and deliver to https://my-agent.example.com/hook.” Acknowledge each dispatch with a 2xx. Delivery is at-least-once, so dedupe on dispatch_id.

See the Alertmanager connector page for the receiver details.

What you get

• One investigation per entity per cooldown window. Repeat notifications for the same API inside the 30 minute cooldown in the example do not start another run.
• Independent handling per service. The cooldown applies to the entity, so a 5xx alert on one API does not suppress a firing on another.
• A timeline in the dispatch. The agent receives the correlated events for that entity over context_window, not just the alert text.
• Resolved alerts as events. With send_resolved: true, each resolved alert arrives with status set to resolved, and you can see it in Recent events on the source.
• A record of every firing and delivery on the trigger’s page, on each dispatch’s page and in the project’s Activity.

The status filter in the example keeps a resolved alert from waking the agent. Filters narrow which events count and which are handed over, so the dispatch timeline then carries the firing events. If you want resolved events in the handed-over timeline, remove that filter. A resolved alert then counts toward the condition like any other event, and the cooldown still limits how often the entity fires.

When something else fits

• If you want to evaluate a metric such as a 5xx rate yourself instead of relying on an Alertmanager rule, use a trigger on a numeric field, for example max(rate_5xx) > 1.0, as described in the triggers guide.
• If you want a periodic summary instead of an alert-driven wake-up, give the trigger a schedule with condition: {every: 10m}.

Related