How do I route Alertmanager alert groups about error spikes to an AI SRE?

Tares routes Alertmanager alert groups to an AI SRE by turning each alert into an event and firing a trigger only on the events that match your error spike symptom. You add an Alertmanager source, filter on the alert name and the firing status, and subscribe an agent to that trigger. Other alerts from the same receiver are still recorded and correlated, but they do not wake the agent.

How it works

Alertmanager has already evaluated, grouped and deduped the alerts. When it POSTs a batch to the source’s ingest URL, Tares fans the alerts array out into one event per alert:

• event_type is the alert name.
• text is the alert’s summary.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including fingerprint, stays in the payload.

The entity key comes from a label automatically (namespace, then service, then job, then instance), or from a primary label you declare. This keeps the alerts on the same timeline as your other sources for that entity.

The symptom filter lives in a trigger. A trigger lists the sources it watches, filters on fields such as event_type and status, and applies a condition over a window with a cooldown. A trigger that matches only your error spike alert name, and only while it is firing, wakes the agent for that one symptom. The agent you subscribe is defined by a prompt, so that prompt is where you tell it the symptom it was woken for. For the full wake-and-investigate loop, see AI SRE.

Set it up

1. Create the source. In the console go to Sources → Add source → Alertmanager alerts, name it and click Create. You can also ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The source page shows a receiver block with the ingest URL filled in. The source looks like this:

- name: am-alerts
  connector: alertmanager

2. Add the receiver to your alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If Tares runs with auth on (tares up --auth), the Alertmanager sender uses a Bearer Authorization header holding an ingest-scoped API key. Without auth on, you do not need it. The source page shows the auth block when the instance requires a token. See Alertmanager connector for the exact block.

4. Add a trigger that names the symptom. Replace HighErrorRate with the alert name your error spike rule uses:

- name: error-spike
  sources: [am-alerts]
  filters:
    - { field: event_type, op: eq, value: HighErrorRate }
    - { field: status, op: eq, value: firing }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 5m

The trigger name is error-spike, so each firing reads as that symptom on the trigger’s page and under Firings. The cooldown keeps a noisy group from waking the agent again right away.

5. Choose who gets woken. Subscribe a Tares agent to the trigger and write its prompt around the symptom, for example: take the first look at an error spike, tie it to specific evidence lines, suggest a next action. Alternatively, point the trigger at your own agent service over a webhook, or at a Slack channel.

6. Verify the filter. Send a test batch, then check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

This alert is not an error spike, so it should appear as an event and the error-spike trigger should not fire. Send the same shape with your error alert’s name and the trigger should fire. Fields shows the alert labels, so you can promote namespace or service to correlate on.

What you get

• Only alerts that match the error spike symptom wake the agent. Other alerts stay on the timeline as context.
• Resolved alerts arrive too when send_resolved is true, and the status filter keeps them from re-waking the agent.
• The trigger name and the agent prompt both state the symptom, so the investigation starts from the right question.
• The woken agent reads one correlated timeline of that entity’s metrics, logs and alerts, not just the alert text.
• Alertmanager keeps owning grouping, silencing and resolved semantics. Tares records what it pushes.

When something else fits

• If you want one trigger for every incident shape, drop the event_type filter and let the agent triage from the alert name. That is the demo setup in the AI SRE guide.
• If your alerts come from a sender other than Alertmanager, use the webhook connector and filter on the fields your payload carries.
• If you want Prometheus rules polled directly instead of Alertmanager’s real-time delivery, the prometheus_alerts connector fits.

Related