How do I wake an AI SRE once when Grafana alert rules report a disk filling up?

Tares wakes your AI SRE once per disk alert when you send Grafana notifications to a webhook source and watch that source with a trigger that has a cooldown. The cooldown is a minimum gap between firings for the same entity, so repeated notifications for the same alert start one investigation. Resolved notifications are still stored on the source, but they do not wake the agent.

How it works

Tares has no connector of its own for Grafana today. Grafana Alerting reaches Tares through the webhook connector and a vendor setup on the Grafana side.

• Grafana posts each notification to the ingest URL of a Tares webhook source. The notification arrives as one event, with the alerts list kept inside the lossless payload.
• Top-level fields such as status, receiver and title work for labels and for text_template. You declare a label for each field you want to filter or group on, and mark one as primary.
• A trigger watches the source. Its filters keep only events whose status is firing, and its condition counts those events in a window.
• cooldown sets the minimum gap between firings for the same entity. The entity is named by key_field, which defaults to the first source’s primary label. While the cooldown runs, further firing notifications for that entity do not fire the trigger again.
• A resolved notification arrives with status set to resolved. Tares stores it on the source like any other event, and the status label shows it in Recent events. The trigger filter does not count it, so it never wakes the agent.

When the trigger fires, Tares POSTs one dispatch to each subscriber, or wakes the Tares agent wired to the trigger in a project. The dispatch carries the entity’s correlated timeline over context_window. For the wider picture of alert-driven investigation, see AI SRE.

Set it up

1. Create a webhook source. Use Sources → Add source → webhook, or ask a connected agent to create it. Declare a label for title (primary) and one for status, then copy the ingest endpoint from the source page. The webhook connector page covers the options.

- name: grafana-alerts
  connector: webhook
  config:
    text_template: '{title}: {status}'
    labels:
      - { name: title, field: title, primary: true }
      - { name: status, field: status }

2. In Grafana, add a Webhook contact point. Set the URL to the Tares ingest URL and the method to POST. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Without auth, leave the header out.
3. In a notification policy, route your disk alert rules to that contact point.
4. Add a trigger that watches the source, counts only firing events, and sets a cooldown. Add it under Triggers → Add trigger or in the catalog YAML. The trigger names the entity with key_field, so each distinct title has its own cooldown. Pick a cooldown longer than the interval at which Grafana repeats the notification.

triggers:
  - name: disk_filling
    sources: [grafana-alerts]
    key_field: title
    filters:
      - { field: status, op: eq, value: firing }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: disk_filling
      context_window: 15m
    cooldown: 30m

5. Wake an agent. Turn the trigger on for a Tares agent in its project, or register an outside agent with the subscribe tool and your own endpoint. The endpoint must be reachable from the Tares server and should answer with a 2xx.
6. Test it by posting a sample to the ingest URL, then check Recent events and Fields on the source page.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"status": "firing", "title": "Disk filling up", "receiver": "tares"}'

Post the same body a second time within the cooldown. The source records both events, and the trigger fires once for that title.

What you get

• One investigation per disk alert instead of one per repeated notification. The cooldown holds back repeat firings for the same entity.
• A dispatch with the entity’s correlated timeline over context_window, so the agent has context without a follow-up call.
• A full record on the source. Firing and resolved notifications are both stored lossless, and Recent events shows each with its status.
• A log of every firing, with per-subscriber delivery status, in the console under Firings and on the trigger’s own page. Delivery is at-least-once, so dedupe on dispatch_id.

When something else fits

• If you want one event per alert instead of one event per notification, use a separate alertmanager source. Grafana’s default payload is Alertmanager compatible, and that source has its own ingest URL.
• If the disk metric already lives in a Prometheus server, you can watch it with a trigger on a numeric field and skip the Grafana hop. See the triggers guide for aggregates such as max.

Related