What should an AI SRE read next to alerts from Prometheus for crash loops?

Tares puts the same service’s metrics, logs and recent deploys on one timeline next to a Prometheus crash loop alert, so an AI SRE reads that evidence and cites it. The alert says that something is wrong. The neighboring events say why, and the diagnosis points at specific lines. You get this by labeling every source with the same service label.

How it works

The prometheus_alerts connector polls Prometheus’s own alerts endpoint, so the alerts your rules have already fired arrive in Tares. Prometheus keeps deciding what counts as a crash loop alert. Tares ingests it and correlates it.

• Each active alert becomes one event. The event_type is the alert name, the text is its summary, and the alert’s labels become Tares labels. Tares adds a state label of firing, pending or resolved.
• State is tracked across polls. A crash loop alert fires once and re-emits only when its state changes. When it clears, you get a resolved event instead of a copy on every poll.
• Sources that carry the same service label merge into one timeline per service, on one clock.
• Next to the alert, an AI SRE reads three kinds of evidence for that service:
  • Metrics from the prometheus connector, polled for the service.
  • Logs from the container through docker_logs, or from your OpenTelemetry pipeline through OTLP.
  • Recent changes, such as deploys and config changes, posted to a webhook source.

The change source is the one worth adding first. Most real root causes are a change, and a deploy event placed just before the alert lets the agent say when the failures started and what landed just before. See Build an AI SRE for the full walkthrough.

Set it up

1. Start Tares and open the console.

   tares up

2. Add the alerts source under Sources, Add source, Prometheus alerts. Enter your Prometheus url and run Discover. You can filter by severity and turn on include pending to also ingest alerts that are still waiting out their for: duration. The source looks like this:

   - name: alerts
     connector: prometheus_alerts
     poll: 30s
     config:
       url: http://localhost:9090
       severities: critical,warning   # blank = all
       labels:
         - { name: severity, field: labels.severity, primary: true }
         - { name: state, field: state }

3. Let a few alerts arrive, then open the source’s Fields tab. An alert’s dynamic labels, such as namespace and service, exist only while it fires, so you promote them from there. Promote service so the alert lands on the right timeline. Prometheus alerts covers the source in detail.

4. Add the evidence sources and give each one the same service label:

   • a prometheus source for the service’s metrics,
   • a docker_logs source for its container logs, or an otlp source,
   • a webhook source for deploys, with a primary service label and event_type set to deploy. Call its ingest URL from your delivery pipeline on every deploy.

5. Open Explore and pick the service. The alert, metrics, logs and deploy events appear merged on one clock. Check that the crash loop alert sits among the others.

6. Connect your agent for on-demand investigation. Start the MCP endpoint in a second shell.

   tares mcp

   Then ask your client what happened to the service in the last 15 minutes. It answers from the correlated read.

7. To wake an agent automatically, create a trigger over the three sources, keyed by service, that fires when an alert event arrives. You choose who is woken: a Tares agent, your own agent service over a webhook, or a Slack channel.

What you get

• One timeline per service, with the crash loop alert beside the metrics, logs and deploys for the same service.
• A diagnosis that cites evidence. The agent ties the failure to specific lines on the timeline, not to the alert text alone.
• Alerts that stay readable. Each alert fires once, and a resolved event marks when it cleared.
• A change in view. A deploy event sits just before the alert, so the agent can name it as a likely cause.
• A loop that learns. The agent writes its finding back onto the service’s timeline, so the next investigation starts from the previous conclusion instead of from scratch.
• Alerting stays where it is. Prometheus keeps owning the rules, and Tares does not re-threshold anything.

When something else fits

• If you route alerts through Alertmanager and want to keep that path, the alertmanager connector takes what it sends to a Tares ingest URL. Use it instead of polling Prometheus.
• If you want to react to the raw numbers instead of a fired alert, write a trigger on the typed metric fields from the prometheus source. The prometheus_alerts source suits teams whose crash loop rules already exist.

Related