How do I find the root cause of 5xx errors on an API with an AI SRE?

Tares gives an AI SRE one correlated timeline per service, with metrics, logs and alerts on one clock, so the agent can name the cause of 5xx errors and cite the evidence lines. A trigger wakes the agent when an alert fires, and the agent writes its finding back onto the service timeline. The next investigation, yours or an agent’s, starts from that conclusion instead of from scratch.

How it works

Tares does the data side and you bring the agent. Three sources feed the timeline, all keyed by the same service label:

• Metrics from Prometheus: 5xx rate, p99 latency, database connections and dependency health.
• Logs from the api-server container’s request logs.
• Alerts that your Prometheus rules fire, such as HighErrorRate.

Because every source carries service, Tares merges them into one timeline per service. Open Explore, pick api-server, and you see the metric that tripped, the error logs and the alert together. What the agent gets shows the exact text an agent reads.

A trigger named incident watches the three sources and fires when an alert event arrives (sum(alert_active) > 0 over 1m, per service). Prometheus keeps owning alerting. Tares ingests what fired, correlates it with the evidence and handles the waking, and nothing is re-thresholded.

The agent, incident-first-look, is subscribed to that trigger. Its definition is a prompt: take the first look, tie the failure to specific evidence lines, suggest a next action. It reads the correlated timeline, names the likely cause, and writes a finding onto the service timeline. You can also pull: a connected MCP client calls read with the service name and answers from the same timeline, including earlier findings.

Set it up

The full walkthrough is in Build an AI SRE. The short version, using the bundled demo as the API to monitor:

1. Have Docker running, plus uv or pipx, and an Anthropic API key so the built-in agent can run.

2. Start the demo system, a small api-server with Prometheus and a traffic generator:

   curl -O https://raw.githubusercontent.com/glassflow/tares/main/demo/docker-compose.yml
   docker compose up -d

3. Install Tares and start it:

   uv tool install tares          # or: brew install glassflow/tap/tares
   tares up

4. Open http://127.0.0.1:8787, go to Projects, pick the AI SRE demo template, paste your Anthropic key and click Start. The three sources, the incident trigger and the incident-first-look agent are created for you.

5. In a second shell, start the MCP endpoint and point your client at it:

   tares mcp

   Then ask: “Use tares: what happened to api-server in the last 15 minutes?”

6. Cause a 5xx spike on purpose with Cause an incident, or from a shell:

   curl -s -XPOST localhost:8080/demo/inject -H 'content-type: application/json' \
     -d '{"scenario": "error_spike"}'

   Within about 30 seconds HighErrorRate fires, the trigger fires, and the agent writes its finding.

7. Point it at your real API. Recreate the three sources against your stack and keep service as the label on all of them. A source for container logs looks like this:

   - name: logs
     connector: docker_logs
     poll: 5s
     config:
       container: tares-demo-api-server   # the container name (docker ps)
       labels:
         - { name: service, const: api-server, primary: true }

   For metrics use the prometheus connector with your own PromQL and service read from the metric labels. For alerts use prometheus_alerts or alertmanager. For logs you can also use OTLP.

8. Add deploys. Most real root causes are a change, so create a webhook source with a primary service label, event_type: deploy and a text template like deploy {commit} by {author}: {message}. Then post one line from your CD pipeline on every deploy:

   curl -X POST https://<your-tares>/ingest/<ingest_key> \
     -H 'content-type: application/json' \
     -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

   A second webhook source with event_type: config covers config changes the same way.

9. Rebuild the trigger for your incident shapes. Keep reacting to fired alerts, or threshold the typed fields directly, such as an error-rate predicate for 5xx storms. Then choose who gets woken: a Tares agent, your own agent service over a webhook, or a Slack channel.

What you get

• A single timeline per service where the 5xx rate, error logs, alerts and deploys sit on one clock, so “deploy landed 14:02, errors started 14:04” is visible at a glance.
• A finding on the service timeline that states what broke, since when, the likely cause tied to evidence lines, and a next action.
• The finding on the project’s Overview under What the agents found, with a headline and the next step. Open it for the full note and how Tares got there.
• A trail you can debug: Activity shows the firing and the run it woke as one thread, and Agents, then incident-first-look, then Runs lists every run.
• A loop that improves. A client you connect later reads the evidence and the earlier finding, so each investigation leaves the timeline smarter.
• Other scenarios to rehearse with: latency and dependency_outage. Clear the fault rolls back, the alerts resolve, and a resolved event lands in the timeline.

When something else fits

• If you already have an agent service, have the trigger POST the timeline to it over a webhook so the investigation starts with zero reads, and keep your own prompts and tooling.
• If you only want to investigate by hand, skip the trigger and use the MCP client with read on the service, which answers from the same correlated timeline.
• If your 5xx errors have no alerting rule yet, add your alert source first, since the default trigger wakes the agent on a fired alert event.

Related