How do I route Alertmanager alert groups about high latency to an AI SRE?

Tares turns every alert Alertmanager sends into its own event, so you can add a trigger that matches only the high latency alert and wakes your agent. Point an Alertmanager receiver at an Alertmanager source in Tares, then filter that source on the alert name and the firing status. Alerts about other symptoms still arrive as events but do not fire the trigger. You name the trigger after the symptom, so the agent you attach to it is the one that handles latency.

How it works

Alertmanager evaluates, groups and dedupes alerts. When it posts a group to the Tares ingest URL, Tares splits the alerts array into one event per alert:

• event_type is the alert name, for example HighLatency.
• text is the alert summary.
• The alert labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including fingerprint, stays in the payload.

Because the alert name is the event type, a trigger filter on event_type selects one symptom. A second filter on status keeps resolved notices from waking the agent. The trigger’s condition counts the matching events in a window, and a cooldown stops a flapping alert from waking the agent repeatedly.

Tares derives the entity key from a label, checking namespace, then service, then job, then instance, unless you declare a primary label. If you want the key to be something specific, set key_label on the source or mark a label as primary.

The agent that wakes is the one subscribed to the trigger. Tares does the data side and you choose who gets woken, as described in Build an AI SRE.

Set it up

1. Create the source. In the console, go to Sources → Add source → Alertmanager alerts, name it and click Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The source page shows the ingest URL.

2. Add the receiver to your alertmanager.yml and reload Alertmanager. Replace the placeholder with your source’s ingest key:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

If your Tares instance has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. Without auth on, you need no key. The Alertmanager connector page shows the exact http_config block.

3. Send a test alert and check Recent events on the source page. The test batch below carries the alert name HighLatency:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"HighLatency","severity":"critical","namespace":"prod"},"annotations":{"summary":"p99 latency is high"},"startsAt":"2024-01-01T00:00:00Z"}]}'

4. Open Fields on the source page and check which labels arrive. Promote the one you want to key on, such as service, so the alert correlates with your other sources.

5. Create a trigger named for the symptom. It reads your Alertmanager source, keeps only firing HighLatency events and fires when at least one arrives in five minutes. Replace alerts with your source name and HighLatency with the alert name in your rules:

- name: high-latency
  sources: [alerts]
  filters:
    - { field: event_type, op: eq, value: HighLatency }
    - { field: status, op: eq, value: firing }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 10m

6. Subscribe your agent to the high-latency trigger. It can be a Tares agent whose prompt covers latency, your own agent service over a webhook, or a Slack channel.

7. Fire a test alert with the curl command from step 3. Check the trigger page for the firing and the agent’s run for the wake-up.

What you get

• Only the symptom you chose wakes the agent. Other alerts from the same Alertmanager route are still recorded as events but do not fire this trigger.
• A trigger whose name says what it covers, so a firing and the agent it wakes are tied to high latency.
• Alertmanager keeps its grouping, silencing and resolved tracking. Tares records what Alertmanager pushes and does not re-threshold anything.
• The full alert, including fingerprint, kept in the event payload.
• A cooldown that limits repeat wake-ups from the same symptom.

To cover another symptom, add another trigger with a different event_type value and attach the agent that suits it.

When something else fits

• If you want the latency decision made on the metric itself, use a trigger over a metrics source and threshold the value there instead of relying on an alert name. The AI SRE guide shows both shapes.
• If the sender is not Alertmanager, such as a custom service or a vendor tool, use the webhook connector and the same trigger pattern over its events.

Related