How do I wake an AI SRE once when Alertmanager alert groups report high latency?

Tares wakes your AI SRE once per entity when you put a cooldown on a trigger that watches your Alertmanager source for the high latency alert. Alertmanager pushes each alert to a Tares ingest URL, and the trigger fires one investigation. Repeats inside the cooldown do not wake the agent again. Resolved alerts still arrive as events with a resolved status.

How it works

Alertmanager groups and dedupes alerts, then POSTs a batch to the ingest URL of an alertmanager source. Tares fans the batch out into one event per alert:

• event_type is the alert name.
• text is the alert summary.
• The alert labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including fingerprint, stays in the payload.

The entity key comes from a label automatically, in the order namespace, service, job, instance. You can change it with key_label or by declaring a primary label.

A trigger watches that source and checks its condition as events are ingested. With group_by: [key_value] the condition is evaluated per entity, so each service fires on its own. The cooldown is the minimum gap between firings for the same entity. A repeating high latency alert for checkout fires one investigation, and the next firing for checkout waits until the cooldown has passed. Another service is not held back by it.

When the trigger fires, Tares delivers the entity’s correlated timeline over context_window to each subscribed agent. The agent starts with context and makes no follow-up call. This is the wake-up flow described in AI SRE.

With send_resolved: true on the receiver, Alertmanager also pushes the resolved notification. It arrives as another event for the same alert with status set to resolved, so the end of the incident is recorded next to the firing event.

Set it up

1. Create the source. Ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Or open Sources → Add source → Alertmanager alerts, name it and select Create. The source page shows a receiver block with the ingest URL filled in.

2. Add the receiver to your alertmanager.yml and reload Alertmanager. Keep send_resolved: true so resolved events reach Tares.

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

If Tares runs with auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. Without auth you need no header. The source page shows the auth block when the instance requires a token.

3. Send a test batch and check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

Fields shows the alert labels, so you can promote namespace or service to correlate on.

4. Define the trigger on the project’s page, under Triggers → Add trigger, or in the catalog YAML. Replace the source name and HighLatency with your own. The filters count only firing events for the high latency alert, the condition fires when at least one arrives in the window, and the cooldown holds repeats back per entity.

triggers:
  - name: high_latency
    sources: [alertmanager_alerts]
    filters:
      - { field: event_type, op: eq, value: HighLatency }
      - { field: status, op: eq, value: firing }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: high_latency
      context_window: 15m
    cooldown: 30m

5. Subscribe the agent. A Tares agent is woken by the trigger in a project that wires it up. An external agent registers a webhook: “Use tares to subscribe to the high_latency trigger and deliver to your endpoint URL.” Your endpoint must be reachable from the Tares server and acknowledge with a 2xx.

What you get

• One investigation per entity for a repeating high latency alert, instead of one per notification.
• Independent firing per service, because the condition is evaluated per entity.
• A timeline in the payload that carries the entity’s correlated events across the trigger’s sources, so the agent starts with context.
• Resolved events from Alertmanager recorded as events with status set to resolved, so the end of an incident is on the record.
• A log of every firing, with per-subscriber delivery status, under Firings, on the trigger’s page and on the project’s Activity.

Deliveries are at-least-once, so dedupe on dispatch_id in your endpoint. A 5xx or transport error is retried with backoff, and a 4xx is recorded as failed and not retried.

Pick a cooldown that is longer than the time your alert takes to repeat, otherwise the same incident can wake the agent again after the cooldown ends. You can read the exact payload of each firing on its dispatch page.

When something else fits

• If you want Tares to poll Prometheus for fired alerts on an interval instead of receiving them, use the prometheus_alerts connector. Alertmanager’s grouping, silencing and resolved semantics, and real-time delivery, favor the alertmanager source.
• If you want a periodic summary instead of a reaction to one alert, give the trigger a schedule such as condition: {every: 10m} and see Watch, then escalate.
• For alerts that come from a numeric metric, such as max(p99_ms) > 1000, use a metric source and aggregate that field in the trigger. The full receiver reference is on the Alertmanager connector page.

Related