How do I stop a flood of Grafana alert rules from waking an AI SRE again and again?

Tares lets you cap a Grafana alert flood with three trigger settings: a cooldown per entity, filters on the labels your source carries, and a key so each entity fires on its own. Grafana reaches Tares through the webhook connector and a Grafana setup, because Tares has no connector of its own for Grafana Alerting today. Your agent is woken once per entity per cooldown window instead of once per notification. This is the watch-and-wake pattern from AI SRE.

How it works

Grafana posts each notification to a webhook source in Tares. The notification arrives as one event, with the alerts list kept inside the lossless payload. Top-level fields such as status, receiver and title work for labels and text_template.

A trigger watches that source and wakes the agent when its condition holds. Three settings control the noise:

• cooldown is the minimum gap between firings for the same entity. A flapping or repeating alert inside that gap does not wake the agent again.
• filters narrow which events count. Each is a field, an op and a value. The field is a label the source carries, or one of event_type, source, text and key_value.
• key_field with group_by: [key_value] names the entity and evaluates the condition per entity, so each one fires independently and has its own cooldown.

If you want one event per alert instead of one per notification, a separate alertmanager source can take the same Grafana payload. The default Grafana payload is Alertmanager compatible, and that source has its own ingest URL.

Set it up

1. In Tares, add a webhook source for Grafana. Declare a label for each payload field you want to filter or key on, and mark one as primary. The fields shown here are top-level fields of the Grafana payload.

- name: grafana-alerts
  connector: webhook
  config:
    text_template: '{status}: {title}'
    labels:
      - { name: title, field: title, primary: true }
      - { name: status, field: status }
      - { name: receiver, field: receiver }

2. Copy the ingest endpoint from the source page. See Inbound webhook for the full source options.
3. In Grafana, add a Webhook contact point with the Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Then route alerts to the contact point from a notification policy.
4. Post a test payload and open the source’s Fields view. It shows which payload fields arrive and how often, so you can pick a reliable key before you rely on it.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"status": "firing", "receiver": "tares", "title": "High error rate"}'

5. Define the trigger. It keys on the primary label, counts events per entity, and waits out a cooldown after each firing. Add the filters you need for severity or environment, for example the env filter shown here, once your source carries that label.

triggers:
  - name: grafana_storm
    sources: [grafana-alerts]
    key_field: title
    filters:
      - { field: env, op: eq, value: prod }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: grafana_storm
      context_window: 15m
    cooldown: 15m

6. Subscribe the agent, or wire a Tares agent to the trigger in a project. Every firing is logged under Firings, so you can check how many wake-ups the cooldown absorbed.

Filters only see labels your source declares. If severity or environment sits inside the alerts list rather than at the top level of the payload, declare it as a label first and confirm in Fields that it arrives. Otherwise the filter has nothing to match.

What you get

• One wake-up per entity per cooldown window, not one per notification.
• Independent firing per key, so a noisy entity does not hide or delay another. Each entity has its own cooldown.
• Narrower triggers, because filters decide which events count and which are handed over.
• Context with every wake-up. The dispatch payload carries the entity’s correlated timeline over the context window, so the agent does not need a follow-up call.
• A record of every firing and delivery in the console, even when a firing has no subscribers.
• Delivery you can trust. It is at-least-once, so dedupe on dispatch_id in your agent.

When something else fits

• If you want one event per alert, use a separate alertmanager source with its own ingest URL instead of the webhook source. It takes the same Grafana payload.
• If you want a periodic summary instead of a condition, give the trigger a schedule such as every: 10m. See Watch with triggers.
• If the problem is how many notifications Grafana sends at all, that is controlled on the Grafana side, in your notification policy and contact point routing.

Related