What data does an AI SRE need to diagnose error spikes?

An AI SRE on Tares needs four signals for error spikes: metrics, logs, alerts, and deploy or config changes, all labeled with the same service. Tares merges them into one timeline per service, and the agent reads that timeline to name a likely cause. Add the deploy source first, because most real root causes are a change.

How it works

Each signal is a source, and every source carries the same service label. That label puts metrics, logs, alerts and deploys on one clock for one service. An agent that starts from this timeline does not spend its first minutes querying each system.

For an error spike, each signal answers a different question:

• Metrics show that the 5xx rate climbed and when it started. Tares polls them from Prometheus. The demo also tracks p99 latency, database connections and dependency health, which help separate an error spike from a latency problem or a failed dependency.
• Logs show what the failing requests looked like. You can bring them from Docker logs, OpenTelemetry (OTLP) or Vercel logs.
• Alerts show what your alerting rules decided. Prometheus alerts or Alertmanager keep owning alerting, and Tares ingests what fired. The trigger that wakes the agent can fire when an alert event arrives.
• Deploys and changes show what changed just before the spike. You can bring them from GitHub or from an inbound webhook that your CD pipeline calls. A second webhook source with event_type: config covers config changes.

When the trigger fires, the agent reads the correlated timeline and writes a finding back onto it. The finding says what broke, since when, the likely cause tied to evidence lines, and a next action. The next investigation, yours or another agent’s, starts from that finding.

The Use AI to monitor your systems guide covers the model in short, and Build an AI SRE walks through it end to end.

Set it up

1. Pick one label. Use service on every source. This is what merges them into one timeline.
2. Add metrics. Create a Prometheus source with your real PromQL, and read service from the metric labels instead of a constant.
3. Add logs. Create a Docker logs source per container, or point your OpenTelemetry pipeline at an OTLP source.
4. Add alerts. Create a Prometheus alerts source or an Alertmanager source. Keep your alerting where it is.
5. Add deploys. Create a webhook source with a primary service label read from the payload’s service field, event_type: deploy, and a text template like deploy {commit} by {author}: {message}. Creating it gives you a stable ingest URL. Then add one line to your CD pipeline, run on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Add a trigger and an agent. Create a trigger over those sources, keyed by service, and subscribe a Tares agent or your own agent service to it. You can fire on alert events, or threshold typed fields directly, such as an error-rate predicate for 5xx storms.
7. Test it. Run tares up, load the AI SRE demo, and use Cause an incident with the error_spike scenario. The alert fires within about 30 seconds and the agent writes its finding onto the timeline.

The full walkthrough is in Build an AI SRE.

What you get

• One timeline per service with metrics, logs, alerts and deploys on the same clock, so the agent sees a sequence such as “deploy landed 14:02, errors started 14:04”.
• A finding on the timeline before anyone opens a dashboard: what broke, since when, the likely cause tied to evidence lines, and a next action.
• The same evidence for interactive work. Claude Code, Cursor and any other MCP client read the same timeline and the earlier findings, so they start from the previous conclusion instead of from scratch.

When the agent cannot tell, look at which signal is missing:

• No deploy or config source. The agent can see that errors rose and where, but has no change to tie them to. This is the most common gap.
• No metrics. There is no clear start time or size for the spike, and nothing to separate it from latency or a dependency problem.
• No logs. The agent knows that errors happened but not what the failing requests looked like, so it has no evidence lines to cite.
• No alerts. The trigger has nothing to react to, so nothing wakes the agent unless you threshold the typed fields yourself.
• Mismatched service labels. The signals stay on separate timelines, and the agent sees each one alone.

When something else fits

If you only want to look at the evidence yourself, you do not need a trigger or an agent. Connect an MCP client and ask it what happened to a service in the last 15 minutes. It reads the same correlated timeline.

If your team already runs its own incident agent, keep it. Tares can POST the timeline to your agent service over a webhook, so the investigation starts with zero reads.

If you want to see the whole loop before touching production, the demo in Use AI to monitor your systems runs on a laptop and lets you cause an error spike on purpose.

Related