What data does an AI SRE need to diagnose 5xx errors on an API?

Tares gives an AI SRE one timeline per service that merges metrics, logs and alerts, and you add deploy events so the agent can tie 5xx errors to a change. Give every source the same service label and the agent starts from the correlated timeline instead of querying each system. Prometheus keeps owning alerting, and Tares correlates what fired with the evidence.

How it works

Each signal is a source, and the shared service label puts the sources on one timeline. For 5xx errors on an API, the AI SRE demo wires three sources and then tells you which signal to add:

• Metrics from the prometheus connector: 5xx rate, p99 latency, database connections and dependency health.
• Logs from the docker_logs connector, which tails the api-server container’s request logs. The otlp connector covers an OpenTelemetry pipeline.
• Alerts from the prometheus_alerts connector, which carries what Prometheus rules fire, such as HighErrorRate. The alertmanager connector does the same for Alertmanager.
• Deploys and config changes from a webhook source that your CD pipeline posts to on every deploy. The demo does not include this source, so you add it for your real systems.

Most real root causes are a change, so the deploy source is the one worth adding first.

When the alert arrives, a trigger watches the sources for that service and wakes a Tares agent, or posts the timeline to your own agent service over a webhook. The agent writes what broke, since when, the likely cause tied to evidence lines and a next action onto the timeline. The Use AI to monitor your systems page covers the same flow in short form.

Set it up

1. Start Tares and open the console at http://127.0.0.1:8787:

tares up

2. Add a prometheus source for your API’s metrics. Use your real PromQL, with service read from the metric labels.
3. Add a log source with docker_logs or otlp, and give it the same service label.
4. Add an alert source with prometheus_alerts or alertmanager. Keep your alerting where it is, and Tares ingests what fires.
5. Add a webhook source for deploys, with a primary service label, a fixed event type and a text template:

- name: deploys
  connector: webhook
  config:
    event_type: deploy
    text_template: 'deploy {commit} by {author}: {message}'
    labels:
      - { name: service, field: service, primary: true }

6. Creating the source gives you a stable ingest URL. Add one line to your CD pipeline, run on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

7. Create a trigger over the sources, keyed by service. You can keep reacting to fired alerts, or threshold the typed fields directly with an error-rate predicate for 5xx storms. Then pick who gets woken: a Tares agent, your own agent service over a webhook, or a Slack channel.

What you get

• A single read of one service’s metrics, logs, alerts and deploys on one clock, so the agent sees a sequence such as “deploy landed 14:02, errors started 14:04”.
• A diagnosis on the timeline before anyone opens a dashboard: what broke, since when, the likely cause tied to evidence lines and a next action.
• The same evidence for you. Claude Code, Cursor and any other MCP client read the same timeline, and they also see the agent’s earlier finding, so an investigation starts from the previous conclusion.

The evidence each source adds shows up as a gap when it is absent:

• No deploy source. The timeline shows the alert, the metric and the logs, but no change event, so the agent has nothing to tie the errors to.
• No log source. The agent sees that the 5xx rate climbed but not the request logs around it.
• No metric source. The alert lands, but the metric that tripped it is not beside it.
• No shared service label. The sources do not land on one timeline, so the agent cannot correlate them.

When something else fits

• If your 5xx errors come from a system with no source above, such as a dependency you do not collect logs from, the agent can only report what the timeline holds. Add a source for it before you expect a root cause.
• If you only want to investigate by hand, connect an MCP client with tares mcp and ask it to read the service’s timeline, with no trigger or agent needed.

Related