How do I find the root cause of a disk filling up with an AI SRE?

Tares gives an AI SRE one correlated timeline per service, so when a disk is filling up the agent reads metrics, logs, alerts and deploys on one clock. It names the likely cause, ties it to specific evidence lines and writes its finding back onto the timeline. The next investigation starts from that conclusion instead of from scratch. The full walkthrough is in AI SRE.

How it works

Tares does the data side and you bring the agent. Every source carries the same service label, so everything about one service merges into a single timeline.

• Metrics come from the prometheus connector, with your own PromQL and service read from the metric labels.
• Logs come from docker_logs per container, or from OTLP if you run an OpenTelemetry pipeline.
• Alerts come from prometheus_alerts or alertmanager. Your alerting stays where it is, and Tares ingests what fires.
• Deploys and config changes come from a webhook source. Most real root causes are a change, so this is the highest-value source to add.

A trigger watches those sources, keyed by service. The demo trigger fires when an alert event arrives, using sum(alert_active) > 0 over 1m per service. Prometheus keeps owning alerting, and Tares correlates what fired with the evidence and wakes the agent. You can also threshold typed fields directly, such as a max(p99_ms) > 1000 style predicate.

When the trigger fires, the agent reads the timeline and writes a finding. The finding says what broke, since when, the likely cause tied to evidence lines, and a next action. It lands on the service’s timeline and on the project’s Overview under What the agents found.

Set it up

1. Install Tares and start it. Add tares mcp in a second shell if you want to investigate interactively from an MCP client.

   uv tool install tares          # or: brew install glassflow/tap/tares
   tares up
   tares mcp        # MCP endpoint on http://127.0.0.1:8788/mcp

2. In the console, go to Sources, Add source and create your metrics, logs and alerts sources against your own stack. Give each one a primary service label so they share a timeline. Alternatively, ask your connected agent to set them up. Point the prometheus source at the PromQL for the metrics you watch, and keep your existing alert rules firing into prometheus_alerts or alertmanager.

3. Add a webhook source for deploys, with a primary service label, event_type: deploy and a text template.

   - name: deploys
     connector: webhook
     config:
       event_type: deploy
       text_template: 'deploy {commit} by {author}: {message}'
       labels:
         - { name: service, field: service, primary: true }

4. Creating the source gives you a stable ingest URL. Add one line to your CD pipeline, run on every deploy.

   curl -X POST https://<your-tares>/ingest/<ingest_key> \
     -H 'content-type: application/json' \
     -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

   A second webhook source with event_type: config covers config changes the same way.

5. Create a trigger over your sources, keyed by service. Keep reacting to fired alerts as the demo does, or write a condition on the typed fields.

6. Pick who gets woken for each trigger:

   • A Tares agent such as incident-first-look. Its whole definition is a prompt, so you edit it for your stack with no infrastructure.
   • Your own agent service over a webhook. Tares POSTs the timeline to your endpoint, so the investigation starts with zero reads.
   • A Slack channel, so every firing and finding lands where your team already looks.

7. Check the result in Explore. Pick the service and you see its metrics, logs, alerts and deploys merged on one clock. What the agent gets shows the exact text the agent reads. Firings and runs appear together under Activity in the project’s Advanced setup.

If you want to try the loop before wiring your own systems, the guide includes a bundled demo system that runs on a laptop with Docker and an Anthropic API key.

What you get

• One timeline per service, so the agent sees a deploy at 14:02 and errors starting at 14:04 in a single read.
• A named likely cause with evidence lines attached, plus a next action.
• A finding written back onto the service’s timeline, with a headline and the full note on the project Overview. Click it to see how Tares got there.
• Compounding investigations. Your connected MCP client reads the evidence and the earlier finding, so each investigation leaves the timeline smarter.
• A trace of every step. The trigger’s firings, the run it woke and every agent run are visible in the console.
• Interactive follow-up. Ask your MCP client something like “what happened to this service in the last 15 minutes?” and it answers from the same correlated read.

When something else fits

If you already run an agent service of your own, route the trigger to it over a webhook and keep your agent logic where it is, since Tares supplies the correlated timeline either way. If you only need a notification when a threshold is crossed, your existing alerting or a Slack delivery on the trigger is enough, and you can add the agent later.

Related