What should an AI SRE read next to Grafana alert rules for a cron job that stopped?

Tares puts the Grafana alert on the same timeline as that service’s metrics, logs and recent changes, so an AI SRE reads all of them together. Grafana reaches Tares through the webhook connector and a vendor setup. Your agent starts from the alert and cites the metric samples, log lines and deploys that sit beside it on the clock.

How it works

Grafana Alerting has no connector of its own in Tares today. It reaches Tares through the webhook connector and a vendor setup: a Grafana Webhook contact point posts each notification to a Tares ingest URL.

The notification arrives as one event. The alerts list stays inside the lossless payload, and top-level fields such as status, receiver and title work for labels and for text_template. The default Grafana payload is Alertmanager compatible. If you want one event per alert, a separate alertmanager source with its own ingest URL can take the same payload.

The alert is only the start. In an AI SRE setup, every source carries the same service label, and Tares builds one timeline per service. For a stopped cron job, an AI SRE reads these next to the alert:

• Metrics. A prometheus source brings the metrics for that service, polled on an interval.
• Logs. A docker_logs source tails the container’s logs, or an OTLP pipeline brings them in.
• Recent changes. A second webhook source with event_type: deploy records each deploy, and another with event_type: config records config changes.
• The job’s own events. A cron job is a producer the webhook connector accepts, so it can post an event of its own and show up on the same timeline.

The agent reads the correlated timeline, either through a trigger that wakes it or through an MCP client you connect. It ties its conclusion to specific evidence lines and writes the finding back to the timeline, so the next investigation starts from it.

Set it up

1. Create a webhook source for the Grafana notifications (Sources → Add source → webhook). Declare a label with the same name your other sources use, such as service, mapped from a top-level field of the notification, and mark one label primary. Save, then copy the ingest endpoint from the source page.
2. In Grafana, add a Webhook contact point. Set its URL to the Tares ingest URL and its method to POST.
3. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Without auth, no header is needed.
4. Route the cron job’s alert rule to that contact point from a notification policy.
5. Check the mapping by posting a sample payload and reading Recent events and Fields on the source page:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "deploy", "sender": "ci", "pipeline": "web"}'

6. Add the evidence sources next to it, all keyed by the same service label: a prometheus source for metrics, docker_logs or OTLP for logs, and a deploy source. Run one line in your CD pipeline on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

7. Add a trigger over those sources and pick who it wakes: a Tares agent, your own agent service over a webhook, or a Slack channel.

What you get

• The Grafana alert lands on the service’s timeline, merged with metrics, logs and changes on one clock.
• A diagnosis that cites evidence: the metric that moved, the log lines around the stop, and the deploy or config change that landed just before it.
• Grafana keeps owning alerting. Tares ingests what fired, correlates it with the evidence and handles the waking, and nothing is re-thresholded.
• A finding written back to the timeline, so the next investigation of the same job starts from the previous conclusion.
• The same evidence available on demand to any connected MCP client, for example with the question “what happened to this service in the last 15 minutes?”

When something else fits

• If you want one event per alert rather than one event per notification, point a separate alertmanager source at the same Grafana payload.
• If your alerts come from Prometheus rules directly, the prometheus_alerts connector polls fired and resolved alerts without a webhook contact point.

Related