How do I route Grafana alert rules about failed deploys to an AI SRE?

Tares takes your failed-deploy Grafana alerts through a dedicated webhook source and a trigger that fires only on that symptom, so unrelated alerts never wake the agent. Grafana sends the notification to the source’s ingest URL, the source tags every event as failed_deploy, and a trigger named failed-deploys fires on firing events of that type. The agent that trigger wakes reads the correlated timeline, where the event type and a readable text line name the symptom.

How it works

Grafana Alerting reaches Tares through the webhook connector and a Grafana contact point. Tares has no connector of its own for Grafana Alerting today, so the setup lives in both products.

The filtering happens in two places:

• In Grafana, a notification policy sends only your failed-deploy alert rules to a webhook contact point that points at one Tares source. Other alerts go to your other contact points.
• In Tares, that source gives every notification a fixed event type, failed_deploy. A trigger filters on that event type and on status being firing, so resolved notifications and anything else on the source do not wake the agent.

A Grafana notification arrives as one event. The alerts list stays inside the lossless payload, and top-level fields such as status, receiver and title are available for labels and for the text template. That is why the example below reads only top-level fields.

The trigger is named for the symptom, so its page and its firings list read as failed deploys. The event it fires on carries the failed_deploy type and a text line built from status and title. The agent reads these on the timeline, so it sees what kind of alert woke it before it looks at anything else.

This is the same pattern as the incident trigger in Build an AI SRE: your alerting stays in Grafana, and Tares ingests what fired, correlates it and handles the waking.

Set it up

1. In Tares, add a webhook source for the symptom (Sources → Add source → webhook) or paste this definition. The fixed event_type is what the trigger filters on.

- name: grafana-failed-deploys
  connector: webhook
  config:
    event_type: failed_deploy
    text_template: '{status} {title}'
    labels:
      - { name: receiver, field: receiver, primary: true }
      - { name: status, field: status }

2. Copy the ingest endpoint from the source page. It looks like /ingest/<ingest_key> on your Tares address.
3. In Grafana, add a Webhook contact point with that ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Without auth on, no header is needed.
4. In Grafana, add a notification policy that routes only your failed-deploy alert rules to this contact point.
5. Check the mapping by posting a small Grafana-shaped payload, then read Recent events and Fields on the source page.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"receiver": "tares", "status": "firing", "title": "Deploy failed"}'

6. Create the trigger. It fires when at least one firing failed-deploy event arrives within five minutes, and the cooldown stops a repeating alert from waking the agent again right away.

- name: failed-deploys
  sources: [grafana-failed-deploys]
  filters:
    - { field: event_type, op: eq, value: failed_deploy }
    - { field: status, op: eq, value: firing }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 5m

7. Choose who gets woken. This can be a Tares agent whose prompt you edit for deploy failures, your own agent service over a webhook, or a Slack channel. The AI SRE guide walks through each choice.

What you get

• Only the alerts you route to the contact point reach the source, and only firing failed_deploy events fire the trigger, so the agent is not woken by unrelated Grafana alerts.
• The trigger is named failed-deploys, so its firings are easy to find and review.
• The event on the timeline carries the failed_deploy type and a text line such as the alert status and title, so the agent can tell what kind of alert woke it.
• The full Grafana payload, including every alert in the notification, is stored lossless on the event.
• The agent works from one correlated timeline and writes its finding back to it, so the next investigation starts from that conclusion.

When something else fits

• If you want one event per alert instead of one per notification, create an alertmanager source with its own ingest URL and point the same Grafana payload at it, since the default Grafana payload is Alertmanager compatible.
• If you want to watch several symptoms at once, add one source and one trigger per symptom, or route everything to one source and filter more finely in the trigger.

Related