How do I wake an AI SRE once when Grafana alert rules report high latency?

Tares wakes an AI SRE once per entity for a high latency alert by receiving Grafana notifications on a webhook source and firing a trigger that has a cooldown. Repeated notifications inside the cooldown do not start another investigation. Resolved notifications are still ingested and stored on the source. Tares has no connector of its own for Grafana Alerting today, so Grafana reaches Tares through the webhook connector and a Grafana-side contact point.

How it works

• Grafana Alerting sends a notification to a Webhook contact point. The webhook connector receives it as one event and stores the payload lossless, with the alerts list inside.
• Top-level fields such as status, receiver and title work for labels and text_template. You mark one label primary, and that label names the entity.
• A trigger watches the source and evaluates its condition as events arrive. With group_by: [key_value] it evaluates per entity.
• cooldown is the minimum gap between firings for the same entity. A repeating high latency alert fires one investigation, and the cooldown absorbs the repeats.
• When the trigger fires, Tares hands the subscribed agent the entity’s correlated timeline over context_window. This is how an AI SRE starts with context instead of polling.
• A filter on status limits which events count and which are handed over. Resolved notifications are still ingested by the source and stored, so you see them under Recent events on the source page.

Set it up

1. Create a webhook source for the Grafana notifications. Mark one label primary. Here receiver names the entity, so the cooldown applies per contact point.

- name: grafana-alerts
  connector: webhook
  config:
    event_type_field: status
    text_template: '{title}'
    labels:
      - { name: receiver, field: receiver, primary: true }
      - { name: status, field: status }

2. Copy the ingest endpoint from the source page. In Grafana, add a Webhook contact point with that Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key.
3. Route the high latency alert rules to that contact point from a notification policy.
4. Post a sample notification and check Recent events and Fields on the source page.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"receiver": "tares", "status": "firing", "title": "High latency", "alerts": [{"status": "firing"}]}'

5. Add a trigger that counts firing notifications per entity and sets the cooldown. Pick a cooldown longer than the gap between repeated notifications you see in Recent events.

triggers:
  - name: high_latency
    sources: [grafana-alerts]
    key_field: receiver
    filters:
      - { field: status, op: eq, value: firing }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: high_latency
      context_window: 15m
    cooldown: 30m

6. Wake the agent. A Tares agent is woken by the trigger in a project that wires it to that trigger, once you turn it on there. An external agent registers a webhook over MCP with the subscribe tool, for example: use tares to subscribe to the high_latency trigger and deliver to your agent’s URL.

If you want one event per alert, and so one entity per alert, point a separate alertmanager source at Grafana instead. It has its own ingest URL, and Grafana’s default payload is Alertmanager compatible.

What you get

• One investigation per entity per cooldown window, however many times Grafana repeats the high latency notification.
• A dispatch with the entity’s correlated timeline over context_window, so the agent has context without a follow-up call.
• A log of every firing, even with zero subscribers. You see it under Firings in the console, on the trigger’s page and in the project’s Activity.
• Resolved notifications kept on the source as stored events, visible under Recent events.
• Delivery that is at-least-once, so dedupe on dispatch_id. Your endpoint acknowledges with a 2xx, and 5xx responses are retried with backoff.

When something else fits

• If you drop the status filter, resolved events are handed over in the dispatch timeline, but they also count toward the condition. Keep the filter if only firing alerts should wake the agent.
• If you want a separate cooldown for each alert instead of each contact point, use an alertmanager source, which makes one event per alert.
• If the alert should be judged on a metric, a Prometheus source with a numeric field and a trigger such as max(p99_ms) > 1000 is a better fit than relaying Grafana’s verdict.

Related