How do I route Alertmanager alert groups about 5xx errors on an API to an AI SRE?

Tares lets you send Alertmanager alerts to an alertmanager source and add a trigger that matches only your 5xx alert, so only those alerts wake the agent. Alertmanager keeps grouping and routing, and Tares ingests each alert in a group as its own event. The trigger filters those events down to the 5xx symptom, and you name the trigger for it.

How it works

• Alertmanager posts batches. The Alertmanager connector receives a batch (alerts: [...]) at the source’s ingest URL and fans it out into one event per alert.
• Each event carries the symptom. The event_type is the alert name, the text is the alert summary, and the alert labels become Tares labels. Tares adds a status label of firing or resolved. The full alert, including its fingerprint, stays in the payload.
• The entity key comes from a label. It is derived from namespace, then service, then job, then instance, or from a label you mark primary. Use key_label to pick the alert label to key on.
• A trigger filters the stream. A trigger lists the source in sources, applies filters on event_type and labels, and checks a condition over a window. Only events that pass all filters count toward the condition, so unrelated alerts such as a database or disk alert never wake the agent.
• The trigger wakes who you choose. In the AI SRE guide, a trigger wakes a Tares agent, your own agent service over a webhook, or a Slack channel. The woken agent reads the correlated timeline for the service.

Set it up

1. Start Tares and create the source. In the console go to Sources → Add source → Alertmanager alerts, name it, and click Create. You can also ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.”

   tares up

2. Add the receiver to your alertmanager.yml with the ingest URL from the source page, then reload Alertmanager.

   receivers:
     - name: tares
       webhook_configs:
         - url: http://localhost:8787/ingest/<ingest_key>
           send_resolved: true
   route:
     receiver: tares

   If your Tares instance runs with auth on, the receiver sends a Bearer Authorization header holding an ingest-scoped API key. The source page shows the exact block to copy. Without auth you need no header.

3. Send a test batch that looks like your 5xx alert, then check Recent events and Fields on the source page. Fields lists the alert labels, so you can see which ones to filter on.

   curl -X POST http://localhost:8787/ingest/<ingest_key> \
     -H 'Content-Type: application/json' \
     -d '{"alerts":[{"status":"firing","labels":{"alertname":"HighErrorRate","severity":"critical","namespace":"prod"},"annotations":{"summary":"5xx rate is high on the API"},"startsAt":"2024-01-01T00:00:00Z"}]}'

4. Create a trigger that keeps only the 5xx alert. Replace HighErrorRate with the name of your own alert rule, and alerts with the name of your source. The event_type filter matches the alert name, and the status filter ignores resolved notifications.

   - name: api-5xx-errors
     sources: [alerts]
     filters:
       - { field: event_type, op: eq, value: HighErrorRate }
       - { field: status, op: eq, value: firing }
     condition: { aggregate: count, predicate: "> 0", window: 5m }
     cooldown: 5m

5. Subscribe the agent you want woken to this trigger, such as a Tares agent whose prompt asks for a first look at the 5xx errors. Use a name that states the symptom, like api-5xx-errors, so its firings are easy to find on the trigger’s page and under Firings.

6. If several services share the alert rule, add one more filter on a label your alerts carry, for example { field: service, op: eq, value: api-server }. Use the label name you saw under Fields. The cooldown keeps repeated notifications from the same incident from waking the agent again within five minutes.

What you get

• Only alerts that match the 5xx symptom wake the agent. Other alerts from the same Alertmanager still land as events on the timeline, but they do not fire this trigger.
• Resolved notifications are recorded as events, since send_resolved is on, but the status filter stops them from waking the agent.
• Alertmanager keeps owning grouping, deduplication and silencing. Tares records what it pushes and correlates it with your other sources, so you do not re-threshold anything.
• Each firing is listed under the trigger’s name, so you can see how often the 5xx symptom woke the agent.
• The woken agent reads the correlated timeline for the service and can write its conclusion back for the next investigation to find. To add deploys to that timeline, add a deploy source as the AI SRE guide describes.

When something else fits

• If you want Tares to poll Prometheus for fired alerts instead of receiving Alertmanager’s pushes, use prometheus_alerts. The Alertmanager connector is the better fit when you want Alertmanager’s grouping, silencing and resolved semantics with real-time delivery.
• If the alerts come from a system that is not Alertmanager, send them to the webhook connector and write the same kind of trigger filters over its labels.

Related