How do I wake an AI SRE once when Grafana alert rules report failed deploys?

Tares wakes your AI SRE once per entity when you send Grafana notifications to a webhook source and put a trigger with a cooldown on it. The trigger counts only firing notifications, and the cooldown holds back repeats for the same entity. Resolved notifications still arrive and are stored on the source, but they do not wake the agent.

How it works

Grafana reaches Tares through the webhook connector and a Grafana-side setup. Tares has no connector of its own for Grafana today. The flow has three parts:

• A Grafana webhook contact point posts each notification to the source’s ingest URL. A notification arrives as one event. The alerts list stays inside the lossless payload, and top-level fields such as status, receiver and title work for labels and text_template.
• A trigger watches the source. A filter on status keeps only firing notifications in the count, and group_by: [key_value] evaluates the condition per entity.
• The cooldown is the minimum gap between firings for the same entity. A failed-deploy alert that Grafana keeps re-sending produces one firing, then silence until the cooldown ends, so one investigation starts instead of many.

Each firing sends one dispatch to the subscribed agent. The dispatch holds the entity’s correlated timeline over the context_window, so the agent starts with context. Filters narrow both which events count and which are handed over, so the hand-over holds the firing notifications that matched. Resolved notifications are still ingested and stored, and you can read them under Recent events on the source page. Every firing is logged on the trigger’s page under Firings, even with zero subscribers.

The entity key is the label that names the entity, so pick a label that stays the same between firing and resolved notifications. receiver does, which makes a contact point used only for failed-deploy alerts a natural fit: its entity is that contact point.

Set it up

1. Create the webhook source. Add it under Sources → Add source → webhook, or through a connected agent. Declare a label per payload field you want to correlate on and mark one primary. The webhook connector page lists the configuration fields. Copy the ingest endpoint from the source page.

- name: grafana-deploys
  connector: webhook
  config:
    text_template: '{status}: {title}'
    labels:
      - { name: receiver, field: receiver, primary: true }
      - { name: status, field: status }

2. Add the contact point in Grafana. Create a Webhook contact point with the Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key.

3. Route the alerts. In a notification policy, route your failed-deploy alert rules to that contact point.

4. Check the mapping. Post a sample notification, then read Recent events and Fields on the source page to see which payload fields arrive.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"receiver": "deploys", "status": "firing", "title": "Deploy failed"}'

5. Define the trigger. Add it on the project’s page, under Triggers → Add trigger, or in the catalog YAML. See Watch with triggers for every field.

triggers:
  - name: failed_deploys
    sources: [grafana-deploys]
    key_field: receiver
    filters:
      - { field: status, op: eq, value: firing }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: failed_deploys
      context_window: 15m
    cooldown: 30m

6. Subscribe the agent. Turn the trigger on in a project that wires your Tares agent to it, or register an outside agent with subscribe(trigger, url) over MCP. Acknowledge each dispatch with a 2xx and dedupe on dispatch_id, because delivery is at-least-once.

Set the cooldown to the longest gap in which a second wake-up would only repeat the first investigation. This is the first place to tune if you get too many or too few wake-ups. Setup for the wider SRE workflow starts at AI SRE.

What you get

• One investigation per failed-deploy entity per cooldown, even when Grafana keeps re-sending the alert.
• A dispatch whose payload is the entity’s correlated timeline over context_window, with the firing notifications that matched your filter.
• Resolved notifications stored on the source with their status, readable under Recent events, without waking the agent.
• A logged firing for every wake-up, with per-subscriber delivery status on the trigger’s page and the exact payload that was sent.

When something else fits

• If you want one event per alert rather than one per notification, a separate alertmanager source can take the same Grafana payload, since the default payload is Alertmanager compatible. It has its own ingest URL.
• If you want to look at the source on a clock instead of reacting to each firing, give the trigger a schedule such as condition: {every: 10m}. It fires once per interval and hands over a summary of the window.

Related