How do I wake an AI SRE once when Grafana alert rules report error spikes?

Tares wakes your AI SRE once per entity when you send Grafana alerts to a webhook source and put a cooldown on the trigger that watches it. Grafana can repeat the same alert many times, and the cooldown turns that into one investigation. Grafana reaches Tares through the webhook connector and a vendor setup, and Tares has no connector of its own for Grafana Alerting today.

How it works

• Grafana posts each notification to the ingest URL of a Tares webhook source. The notification arrives as one event, with the alerts list kept inside the lossless payload.
• Top-level fields such as status, receiver and title work for labels and for text_template.
• A trigger watches the source, evaluates its condition as events arrive, and with group_by: [key_value] evaluates it per entity. The entity is the value of the trigger’s key_field.
• When the condition holds, the trigger fires and wakes the agent that is wired to it, handing over the entity’s correlated timeline over context_window.
• cooldown is the minimum gap between firings for the same entity. While it runs, repeated Grafana notifications for that entity do not wake the agent again.

The trigger in this article has a filters entry on status. Filters narrow which events count and which are handed over. A notification whose status is not firing does not count toward the condition and is not part of this trigger’s dispatch. Every notification is still stored as an event on the source, and you can read it under Recent events on the source page.

Set it up

1. In Grafana, add a Webhook contact point with the Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key.
2. Route your error alert rules to that contact point from a notification policy.
3. In Tares, add a webhook source under Sources → Add source → webhook, or in the catalog YAML. Mark one label primary. Copy the ingest endpoint from the source page. See Inbound webhook for the connector.

- name: grafana_alerts
  connector: webhook
  config:
    text_template: '{status} {title}'
    labels:
      - { name: receiver, field: receiver, primary: true }
      - { name: status, field: status }
      - { name: title, field: title }

4. Post a sample notification to check the mapping, then read Recent events and Fields on the source page. A push source’s test_source only returns the ingest endpoint, so it does not check labels.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"receiver": "tares", "status": "firing", "title": "error spike"}'

5. Add a trigger with a cooldown, under Triggers → Add trigger or in the catalog YAML. The count aggregate needs no numeric field. Details are in Watch with triggers.

triggers:
  - name: grafana_error_spike
    sources: [grafana_alerts]
    key_field: receiver
    filters:
      - { field: status, op: eq, value: firing }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: error_spike
      context_window: 15m
    cooldown: 30m

6. Wire the agent to the trigger. A Tares agent is woken by the trigger in a project that wires it, once you turn it on there. An external agent subscribes over MCP:

Use tares to subscribe to the grafana_error_spike trigger and deliver to https://my-agent.example.com/hook.

Tares POSTs one JSON body per firing. Acknowledge it with a 2xx, and dedupe on dispatch_id, because delivery is at-least-once.

What you get

• One investigation per entity per cooldown window, however often Grafana repeats the alert. Here the entity is the receiver value, so each contact point name is its own entity.
• A dispatch whose payload is the entity’s correlated timeline over context_window, so the agent has context without a follow-up call.
• A record of every firing, even with zero subscribers, under Firings in the console, on the trigger’s page and on the project’s Activity.
• Every Grafana notification stored on the source, including ones the trigger’s filter skips, so you can still inspect them under Recent events.

This fits the AI SRE flow: the alert wakes the agent once, and the agent starts from the timeline instead of polling.

When something else fits

• If you want resolved notifications in the timeline the agent receives, remove the status filter from the trigger. Resolved notifications then count toward the condition too, so check that the cooldown still gives you one wake-up.
• If you want one event per alert instead of one per notification, use a separate alertmanager source with its own ingest URL. The default Grafana payload is Alertmanager compatible, so it can take the same payload.

Related