What should an AI SRE read next to Grafana alert rules for crash loops?

Next to the Grafana alert rule, an AI SRE reads the same service’s metrics, logs and recent deploys on one Tares timeline. The alert is where the investigation starts, and the other signals sit beside it on the same clock. The diagnosis can then cite specific evidence lines instead of repeating the alert text.

How it works

Tares has no connector of its own for Grafana Alerting today. The alert reaches Tares through the webhook connector and a Grafana-side setup: a Webhook contact point that POSTs to a Tares ingest URL.

The timeline comes from one label. Every source for a service carries a primary service label with the same value. Sources that share it merge into one timeline for that service, as described in Build an AI SRE.

For a crash loop, the agent reads four things on that timeline:

• The Grafana alert event. One event per notification. The full payload is stored lossless, including the alerts list. Top-level fields such as status, receiver and title can be labels or go into text_template.
• The service’s metrics. A prometheus source with the restart, health or error metrics you pick by name or label.
• The service’s logs. A docker_logs source, or OTLP if you run OpenTelemetry, so the crash output sits next to the alert.
• Recent changes. A webhook source that your deploy pipeline POSTs to on every deploy. Most crash loops start with a change, so this is the signal most likely to explain the alert.

Because receiver is the same for every alert sent through one contact point, it cannot identify the service. Instead, create one Grafana webhook source per service and give it a constant service label.

Set it up

1. Create a webhook source for the service, here checkout, with a constant primary service label. Keep receiver and status as plain labels.

- name: checkout-grafana-alerts
  connector: webhook
  config:
    text_template: '{title} {status}'
    labels:
      - { name: service, const: checkout, primary: true }
      - { name: status, field: status }
      - { name: receiver, field: receiver }

2. Copy the ingest URL from the source page. In Grafana, add a Webhook contact point with that URL and the POST method. Route your crash loop alert rules to the contact point from a notification policy. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Repeat steps 1 and 2 for each service, each with its own source and ingest URL.

3. Add the change signal. Create a second webhook source for deploys, with service read from the payload.

- name: checkout-deploys
  connector: webhook
  config:
    event_type: deploy
    text_template: 'deploy {commit} by {author}: {message}'
    labels:
      - { name: service, field: service, primary: true }

Then add one line to your CD pipeline, run on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

4. Give the metrics and logs sources the same service label. On the prometheus source, read it from the metric labels with { name: service, field: metric.service, primary: true }. On a docker_logs source, set it as a constant with { name: service, const: checkout, primary: true }.

5. Connect your MCP client and ask about the service. The client calls read for checkout and answers from the merged timeline.

tares mcp

Then ask: “Use tares: what happened to checkout in the last 15 minutes?”

What you get

• One timeline per service where the Grafana alert, the metrics, the logs and the deploys appear in time order.
• A diagnosis that ties the crash loop to evidence, for example a deploy that landed minutes before the alert fired and the log lines that followed it.
• Grafana keeps owning alerting. Tares ingests what fired, correlates it with the evidence and does not re-threshold anything.
• Findings the agent writes back to the timeline, so the next investigation of the same service starts from the previous conclusion.

When something else fits

• If you want one Tares event per alert instead of one per notification, use a separate alertmanager source with its own ingest URL. Grafana’s default payload is Alertmanager compatible, so it can take the same payload.
• If the alerts come from Prometheus rules rather than Grafana, the prometheus_alerts connector polls them directly.
• If Grafana alerts are all you have, with no metrics, logs or deploys beside them, the agent only has the alert text to cite, so add at least one more source for the service.

Related