What data does an AI SRE need to diagnose high latency?

Tares merges metrics, logs, alerts and deploy events for a service into one timeline that an AI SRE reads when you investigate high latency. The AI SRE guide builds on metrics, logs and alerts, then adds deploys and config changes as the source most worth connecting next. Every source carries the same service label, so the agent reads one correlated view instead of querying each system.

How it works

Each signal is a source in Tares, and the shared service label is what puts the sources on one timeline. For a latency symptom, the signals map like this:

• Metrics. The prometheus connector polls your latency series, such as p99. The demo’s metrics source carries p99, 5xx rate, DB connections and dependency health.
• Logs. The docker_logs connector tails a container’s request logs. The otlp connector or a Vercel log drain also carry logs.
• Alerts. The prometheus_alerts or alertmanager connector ingests what your rules fire. The demo’s Prometheus has a HighLatency rule. Prometheus keeps owning alerting, and Tares ingests what fired.
• Deploys and config changes. A webhook source that your CD pipeline posts to on every deploy. A second webhook source with event_type: config covers config changes.

When the signals share a service, an alert lands next to the metric that tripped it and the logs around it. A trigger watches those sources and wakes a Tares agent, or your own agent service over a webhook. The agent starts with the correlated timeline already in front of it and writes a finding onto the timeline: what broke, since when, the likely cause tied to evidence lines, and a next action. The AI monitoring guide describes the same flow.

Set it up

1. Start Tares and open the console at http://127.0.0.1:8787.

tares up

2. Add a metrics source with the prometheus connector. Point its PromQL at your latency series and read service from the metric labels.
3. Add a log source, either docker_logs per container or otlp from your OpenTelemetry pipeline. Give it the same service label.
4. Add an alert source, either prometheus_alerts or alertmanager. Keep your alerting where it is.
5. Add the deploy source. Create a webhook source with a primary service label read from the payload’s service field, event_type: deploy, and a text template like deploy {commit} by {author}: {message}. Creating it gives you a stable ingest URL. Then add one line to your CD pipeline, run on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Build a trigger over these sources keyed by service. You can keep reacting to fired alerts, or threshold the typed fields directly, for example max(p99_ms) > 1000 for latency.
7. Pick who gets woken: a Tares agent, your own agent service over a webhook, or a Slack channel.

The Build an AI SRE guide walks through the same steps against a demo system first. Its latency scenario lets you cause a latency incident on purpose.

What you get

• One timeline per service with latency metrics, request logs, alerts and deploys on one clock.
• A deploy event on the timeline, so the agent sees “deploy landed 14:02, errors started 14:04” instead of guessing at a cause.
• A finding written back to the timeline, which the next investigation starts from, yours or another agent’s.
• The same evidence through MCP, so Claude Code, Cursor or another MCP client can answer “what happened to this service in the last 15 minutes?” from the correlated read.

When the agent cannot tell, look for the gap in the sources:

• No deploy or config source. The timeline holds no change events, so the agent cannot line a latency rise up with a change. Most real root causes are a change, so this is the gap to close first.
• No alert source. A trigger that fires on an alert event, like the demo’s incident trigger, has nothing to react to. Threshold a metric field in the trigger instead.
• No metrics source. The agent sees alerts and logs but not the latency curve that tripped them.
• No logs source. The agent sees the numbers but not the requests behind them.
• A source without the shared service label. It stays off that service’s timeline, so the agent never sees it next to the rest.

When something else fits

• If you want to investigate by hand with no trigger, connect an MCP client with tares mcp and ask it about the service. It reads the same timeline.
• If your latency data lives in a system with no listed connector, the http_poll connector can poll a JSON endpoint on a schedule, and the webhook connector accepts arbitrary JSON, so you can bring that data in either way.

Related