How do I route Grafana alert rules about 5xx errors on an API to an AI SRE?

Tares routes Grafana alert rules about 5xx errors to an AI SRE through a dedicated webhook source, a Grafana notification policy and a trigger. Only the rules you send to that contact point reach the source, so only 5xx alerts wake the agent. The source and trigger are named for the symptom, so the agent starts from “5xx errors on the API” and not from a generic alert.

How it works

Grafana Alerting reaches Tares through the webhook connector and a vendor setup. Tares has no connector of its own for Grafana Alerting today.

The filtering to one symptom happens in three places:

• Grafana notification policy. You create a Webhook contact point for this symptom and route only your 5xx alert rules to it. Other rules never reach this Tares source.
• A source for this symptom only. A webhook source with a fixed event_type and a text_template that names the symptom. The notification arrives as one event, and the full alerts list stays inside the lossless payload. Top-level fields such as status, receiver and title work for labels and text_template.
• A trigger on that source. The trigger watches only this source, so it fires only for the 5xx alerts that source received. It wakes the agent subscribed to it. The trigger name, the event type and the event text all say the symptom.

For how triggers, sources and agents fit together end to end, see AI SRE. For the connector itself, see Inbound webhook.

Set it up

1. Create the webhook source. Go to Sources → Add source → webhook, or ask a connected agent to create it. The fixed event_type and the text_template carry the symptom into the event the agent reads. Save it and copy the ingest endpoint from the source page.

- name: grafana-api-5xx
  connector: webhook
  config:
    event_type: api_5xx_alert
    text_template: 'API 5xx errors: {title} is {status}'
    labels:
      - { name: title, field: title, primary: true }
      - { name: status, field: status }

2. Add the contact point in Grafana. Add a Webhook contact point, set the URL to the Tares ingest URL and the method to POST. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Without auth on, you need no header.

3. Route only the 5xx rules to it. In a notification policy, send the alert rules about 5xx errors on your API to this contact point and nothing else. This is the step that narrows the stream to one symptom.

4. Create the trigger. Watch only the 5xx source. The count condition fires when at least one notification arrives in the window, and the cooldown stops a burst of notifications from waking the agent repeatedly.

- name: api-5xx-errors
  sources: [grafana-api-5xx]
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 5m

5. Subscribe an agent. Subscribe a Tares agent to api-5xx-errors. A Tares agent is defined by its prompt, so write the symptom into it, for example to take the first look at 5xx errors on the API and tie the failure to specific evidence lines. You can also deliver the firing to your own agent service over a webhook, or to a Slack channel.

6. Test the path. Post a sample notification to the ingest URL, then open Recent events on the source page. Fields shows which payload fields arrive and how often.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"receiver": "tares", "status": "firing", "title": "HighApi5xx", "state": "alerting"}'

What you get

• Only alert rules you route to the contact point reach the source, so only 5xx alerts can fire the trigger.
• Each event carries the symptom in its event type and its text, for example API 5xx errors: HighApi5xx is firing. The status shows in the text, so the agent sees whether the alert is firing or resolved.
• The full Grafana payload, including the alerts list with labels, annotations and timestamps, is stored lossless on the event.
• The trigger name, api-5xx-errors, tells you at a glance what woke the agent when you read the firings.
• Grafana keeps owning alert evaluation. Tares ingests what Grafana fires and handles the waking.

When something else fits

• If you want one event per alert and not one per notification, point a separate alertmanager source at the same Grafana payload. The default Grafana payload is Alertmanager compatible, and that source has its own ingest URL.
• If you want all Grafana alerts to wake the agent, skip the dedicated contact point and route every rule to one webhook source instead.
• If your 5xx alerts come from Prometheus rules, the prometheus_alerts connector reads them directly, without Grafana in the path.

Related