How do I find the root cause of error spikes with an AI SRE?

Tares gives your AI SRE one correlated timeline per service, so the agent can name the cause of an error spike, cite evidence lines and write its finding back. Metrics, logs, alerts and deploys that share a service label merge on one clock. A trigger wakes the agent when an alert fires, and the finding it records is there for the next investigation.

How it works

Tares handles the data side and you bring the agent. The full walkthrough is in Build an AI SRE.

• One timeline per service. Every source carries the same service label. Tares merges the metrics, request logs and alerts for that service on one clock, so a 5xx climb sits next to the error logs and the alert that reported it.
• A trigger wakes the agent. Prometheus keeps owning alerting. Tares ingests what fired, correlates it with the evidence and fires a trigger, for example when an alert event arrives for a service. Nothing is re-thresholded.
• The agent reads one correlated view. A Tares agent, or your own agent over a webhook or MCP, starts from the whole timeline instead of querying each system separately.
• The agent names the cause and cites evidence. Its job is to say what broke, since when, tie the failure to specific evidence lines and suggest a next action.
• The finding is written back. The conclusion lands on the service’s timeline. The next investigation, by you or by an agent, reads the evidence and the earlier finding together, so it starts from the previous conclusion.

Set it up

1. Install Tares and start it.

   uv tool install tares          # or: brew install glassflow/tap/tares
   tares up

2. Create three sources for the service you want to watch, all keyed by the same service label. Use prometheus style metrics with your real PromQL and service read from the metric labels, logs from docker_logs or OTLP, and alerts from prometheus_alerts or alertmanager. Add them in the console under Sources, then Add source. To try it first, pick the AI SRE demo template under Projects.

3. Add the signal most real error spikes trace back to, which is a change. Create a webhook source with a primary service label, event_type: deploy and a text template such as deploy {commit} by {author}: {message}. Creating it gives you an ingest URL. Add one line to your CD pipeline so every deploy posts to it:

   curl -X POST https://<your-tares>/ingest/<ingest_key> \
     -H 'content-type: application/json' \
     -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

   A second webhook source with event_type: config covers config changes the same way.

4. Create a trigger over those sources, keyed by service. Keep reacting to fired alerts, or threshold typed fields directly, such as an error-rate predicate for 5xx storms.

5. Choose who gets woken per trigger. Pick a Tares agent whose whole definition is a prompt (take the first look, tie the failure to evidence lines, suggest a next action), your own agent service over a webhook, or a Slack channel.

6. Start the MCP endpoint in a second shell and point your MCP client at it, so you can investigate the same timeline on demand.

   tares mcp        # MCP endpoint on http://127.0.0.1:8788/mcp

7. Test the chain with the demo system. Cause an error spike from the project page with the Cause an incident button, or from a shell:

   curl -s -XPOST localhost:8080/demo/inject -H 'content-type: application/json' \
     -d '{"scenario": "error_spike"}'

   Within about 30 seconds the 5xx rate climbs, the alert fires, the trigger fires and the agent writes its finding. Use the scenario clear to roll back, and the alerts resolve.

What you get

• A named cause with evidence. The finding says what broke, since when and the likely cause, tied to evidence lines on the timeline.
• A view you can read yourself. Open Explore, pick the service and see metrics, logs and alerts merged on one clock. The timeline turns red when the alert lands.
• Deploys in the same picture. With the webhook source in place, the agent sees a deploy land at 14:02 and errors start at 14:04.
• The diagnosis before you look. Nothing is polled by the agent and nobody has to page it. The finding appears on the project Overview under what the agents found, with a headline and the next step.
• An audit trail. The trigger’s firings and every agent run are listed in the console, so you can see how Tares got to the conclusion.
• A loop that improves. Every investigation leaves a finding on the timeline. Your connected MCP client reads that finding along with the evidence, so the next question starts from the previous answer.

When something else fits

• You only need a page when a threshold is crossed. Keep that in Prometheus or Alertmanager. Tares reads what fires and adds the correlation and the waking.
• You want to investigate by hand. The MCP client path on its own is enough. Ask what happened to the service in the last 15 minutes and the client answers from the correlated read, with no trigger or agent needed.
• You already run your own agent service. Have the trigger POST the timeline to your endpoint over a webhook. Your agent starts with the evidence already in hand and writes its finding back.
• Your services share no common label. The per-service timeline depends on every source carrying the same service label, so add that label to each source first.

Related