How do I route Grafana alert rules to an AI SRE?

Tares receives Grafana alert rules through a webhook source and a Grafana Webhook contact point, then a trigger wakes your agent with the correlated timeline. Tares has no connector of its own for Grafana Alerting today, so Grafana reaches Tares through the webhook connector and a vendor setup. You create the webhook source, point a Grafana contact point at its ingest URL, and add a trigger over that source.

How it works

Grafana sends each notification as one JSON POST to the ingest URL of a Tares webhook source. Tares stores the payload lossless, so the full alerts list stays available to the agent.

• The default Grafana payload carries top-level fields such as status, receiver and title. Those fields work for labels and for text_template on the source.
• The alerts list arrives inside the same event, so one notification is one event.
• A trigger watches the source and fires when its condition holds. It wakes the agent you chose: a Tares agent, your own agent service over a webhook, or a Slack channel.
• The agent reads one correlated timeline instead of a bare alert, and writes its finding back where the next investigation finds it.

The webhook connector page covers the source in detail. The AI SRE guide shows the full loop of sources, triggers and agents, with Prometheus and your own alerting staying where they are while Tares ingests what fires.

Set it up

1. Create a webhook source in Tares under Sources → Add source → webhook, or ask a connected agent to create it. Use top-level payload fields for the labels and mark one as primary.

- name: grafana-alerts
  connector: webhook
  config:
    event_type_field: status
    text_template: '{title}: {status}'
    labels:
      - { name: receiver, field: receiver, primary: true }
      - { name: status, field: status }

2. Copy the ingest endpoint from the source page. It has the form /ingest/<ingest_key>.
3. In Grafana, add a Webhook contact point. Set its URL to the Tares ingest URL and its method to POST.
4. If Tares runs with auth on, set the contact point’s Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Creating a push source in the console mints a key for it and shows it once. Without auth on, you need no header.
5. Route alerts to the contact point from a notification policy in Grafana.
6. Check the mapping by posting a sample payload to the ingest URL, then open Recent events and Fields on the source page.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"receiver": "tares", "status": "firing", "title": "HighErrorRate"}'

7. Add a trigger over the source. It counts matching events in a window and fires when the count passes the predicate. Then subscribe your agent to it.

- name: grafana-firing
  sources: [grafana-alerts]
  filters:
    - { field: status, op: eq, value: firing }
  condition: { aggregate: count, predicate: "> 0", window: 5m }

If you want one event per alert instead of one per notification, create a separate alertmanager source with its own ingest URL and send the same Grafana payload to it. The default Grafana payload is Alertmanager compatible.

What you get

• Grafana alert rules land in Tares as events on a timeline, next to your metrics, logs and deploys.
• A firing alert wakes your agent through a trigger, and the agent starts from the correlated timeline instead of from zero reads.
• The full notification, including the alerts list, is kept lossless, so nothing Grafana sent is dropped.
• The agent’s conclusion is written back to the timeline, so the next investigation starts from it.
• Grafana keeps owning alerting. Tares ingests what fired and handles the waking.

When something else fits

• If your alerts come from Prometheus rules or Alertmanager, use the prometheus_alerts or alertmanager connector, which Tares provides directly, rather than going through Grafana.
• If you need a single event per alert with its own labels, use an alertmanager source with the Grafana payload instead of the webhook source.

Related