How do I find the root cause of high latency with an AI SRE?

Tares puts metrics, logs, alerts and deploys for each service on one timeline, so an AI SRE can name the cause of high latency from evidence. A trigger wakes the agent when a latency alert fires, or you ask your own MCP client on demand. The agent ties its conclusion to specific evidence lines and writes a finding back to the service’s timeline. The next investigation starts from that finding instead of from scratch. The full build is in Build an AI SRE.

How it works

Tares handles the data side and you bring the agent. Every source carries the same service label, so Tares merges all of them into one timeline per service on one clock.

• Metrics. The prometheus connector polls things like the p99 latency, the 5xx rate and database connections.
• Logs. The docker_logs connector tails a container’s request logs. OTLP works too if you run an OpenTelemetry pipeline.
• Alerts. The prometheus_alerts or alertmanager connector ingests what your own rules fire, such as HighLatency. Prometheus keeps owning alerting. Tares ingests what fired and correlates it with the evidence. Nothing is re-thresholded.
• Deploys and config changes. A webhook source records each deploy as an event on the service’s timeline.

Two paths reach the agent:

• Push. A trigger such as incident watches the sources per service and fires when an alert event arrives. It wakes a Tares agent, for example incident-first-look. That agent reads the correlated timeline, names what broke and since when, ties the likely cause to evidence lines and suggests a next action. Then it writes a finding onto the service’s timeline.
• Pull. You connect an MCP client and ask a question. The client calls read for the service and answers from the same correlated timeline, including earlier findings.

Set it up

1. Install Tares and start it. Use uv tool install tares or brew install glassflow/tap/tares, then run tares up. Open the console at http://127.0.0.1:8787.

2. Create the sources. Under Sources, add a metrics source, a logs source and an alerts source. Give all three the same primary service label. The AI SRE demo project template under Projects creates this wiring for you, with three sources, the incident trigger and the incident-first-look agent. The built-in agent needs an Anthropic API key, which you paste on the setup page or add under Settings, Model providers.

3. Add deploys. Most real latency causes are a change. Create a webhook source with a primary service label, event_type: deploy and a text template such as deploy {commit} by {author}: {message}. Creating it gives you a stable ingest URL. Post to that URL from your CD pipeline on every deploy. Each post lands as a deploy event, so the agent sees a deploy landing at 14:02 and latency climbing at 14:04.

4. Set the trigger. Keep reacting to fired alerts such as HighLatency, or threshold the typed fields directly, for example max(p99_ms) > 1000 for latency. Choose who gets woken: a Tares agent with a prompt you edit for your stack, your own agent service over a webhook, or a Slack channel.

5. Connect your MCP client. Start the MCP endpoint in a second shell and point your client at it.

tares up
tares mcp        # MCP endpoint on http://127.0.0.1:8788/mcp

6. Ask. For example: “Use tares: what happened to api-server in the last 15 minutes?” To rehearse first, run the demo’s latency scenario with the Cause an incident button on the project page. Within about 30 seconds the alert lands on the timeline, the trigger fires and the agent writes its finding. Clear the fault afterwards and the alerts resolve.

What you get

• One read instead of many tabs. The agent works from metrics, logs, alerts and deploys merged per service, so a latency spike sits next to the log lines and the deploy around it.
• A named cause with evidence. The finding says what broke, since when, the likely cause tied to evidence lines, and a next action.
• Findings where you look. The finding appears on the project’s Overview under What the agents found, and on the service’s timeline in Explore. A Slack channel can receive every firing and finding too.
• A debuggable chain. Activity shows the firing and the run it woke as one thread, and each agent’s Runs page lists every run.
• A timeline that gets smarter. Your connected client reads the evidence and the earlier finding, so each investigation starts from the last conclusion.

When something else fits

• If the cause sits in a system you have not connected, the agent cannot cite it. Add a source for it, keyed by the same service label, before you rely on the finding.
• If you already run your own agent service, let the trigger POST the timeline to it over a webhook instead of using the built-in agent. The investigation then starts with zero reads.

Related