How do I find the root cause of crash loops with an AI SRE?

Tares gives your agent one correlated timeline per service, so it can name the cause of a crash loop from metrics, logs, alerts and deploys on one clock. The agent cites the evidence lines behind its conclusion and writes a finding back onto the same timeline. The next investigation starts from that finding instead of from scratch. The full walkthrough is in Build an AI SRE.

How it works

Tares handles the data side and you bring the agent. You label every source with the same service label, and Tares merges everything that carries it into one timeline per service.

• Metrics, logs and alerts share a clock. A metric source polled from Prometheus, a log source from the container, and an alert source all land on the same service timeline. A crash loop shows up as the alert, the log lines and the metric movement side by side.
• Deploys and config changes join the timeline. A webhook source with event_type: deploy records each change. Most real root causes are a change, so the agent can see that a deploy landed and the failures began shortly after.
• Prometheus keeps owning alerting. Tares ingests the alerts your rules fire, correlates them with the evidence and wakes the agent. Nothing is re-thresholded.
• A trigger wakes the agent. The trigger watches your sources, keyed by service, and fires when an alert event arrives. The agent reads the correlated timeline and writes a finding.
• The finding is written back. It states what broke, since when, the likely cause tied to evidence lines, and a next action. It appears on the service timeline and in the project overview.

You can also pull instead of push. Connect an MCP client and ask it what happened to the service. It reads the same timeline and the earlier finding.

Set it up

1. Install Tares and start it. You need Docker for the demo system, uv or pipx for the install, and an Anthropic API key for the built-in agent.

   tares up

2. Open the console at http://127.0.0.1:8787, go to Projects and pick the AI SRE demo template to see the wiring working end to end. The demo has three sources keyed by service, one trigger named incident, and one agent named incident-first-look.

3. Recreate the sources against your own stack, all keyed by the same service label:

   • Metrics with the prometheus connector, reading service from your metric labels.
   • Logs with the docker_logs connector, or with OTLP if you run an OpenTelemetry pipeline.
   • Alerts with the prometheus_alerts or alertmanager connector. Keep your alerting where it is.

4. Add the deploy source, because it is the highest-value signal the demo lacks. Create a webhook source with a primary service label, event_type: deploy, and a text template built from the commit, author and message. Creating it gives you a stable ingest URL, and your CD pipeline posts to it on every deploy.

   - name: deploys
     connector: webhook
     config:
       event_type: deploy
       text_template: 'deploy {commit} by {author}: {message}'
       labels:
         - { name: service, field: service, primary: true }

   A second webhook source with event_type: config covers config changes the same way.

5. Build a trigger over those sources, keyed by service. Either react to fired alerts as the demo does, or threshold the typed fields directly.

6. Choose who gets woken per trigger. Use a Tares agent and edit its prompt for your stack, or send the timeline to your own agent service over a webhook, or post to a Slack channel.

7. Start the MCP endpoint in a second shell and connect your client, so you can investigate on demand.

   tares mcp

   Then ask: what happened to your service in the last 15 minutes? The client calls read with the service name and answers from the correlated read.

What you get

• A named cause with evidence. The agent ties the failure to specific lines on the timeline, such as the alert, the log output and the deploy, so you can check its reasoning.
• A diagnosis before you look. When the trigger fires, the agent runs on its own. Nobody pages it and nothing is polled by hand.
• A timeline that gets smarter. Each finding is written back to the service timeline. A later investigation, from the built-in agent or your MCP client, reads the evidence and the previous conclusion together.
• An audit trail. The console shows the trigger firing and the run it woke as one thread, and every agent run is listed under the agent.
• One read per service. The agent works from a single correlated view instead of querying each system on its own.

When something else fits

• If you only want to chat with your data, skip the trigger and agent. Connect an MCP client and ask it questions against the timeline.
• If you already run an agent service, keep it. Tares can POST the correlated timeline to your endpoint over a webhook, so the investigation starts without extra reads.
• If a service has no shared service label across its sources, fix that first, because the per-service timeline depends on it.

Related