Build an AI SRE

Stand up a system to monitor

An SRE needs a production to watch. The demo stack stands in for yours: a small api-server (Prometheus metrics, request logs, and a fault switch you’ll use to cause incidents), Prometheus scraping it and evaluating three alerting rules (HighErrorRate, HighLatency, DependencyDown), and a traffic generator. One compose file, no checkout:

curl -O https://raw.githubusercontent.com/glassflow/tares/main/demo/docker-compose.yml
docker compose up -d

Give it ~10 seconds, then confirm it’s alive:

curl -s localhost:8080/api/stats                  # {"ok": true, ...}
curl -s 'localhost:9090/api/v1/query?query=up'    # prometheus is scraping

Wire Tares to it

Install Tares, then wire it to the stack one of two ways. They are separate paths, not two halves of one: the console path starts a plain tares up and does the rest from the browser; the catalog path starts Tares already seeded, and only seeds an empty instance. Pick one.

uv tool install tares          # or: pipx install tares

tares up

curl -O https://raw.githubusercontent.com/glassflow/tares/main/demo/catalog.demo.yaml
export ANTHROPIC_API_KEY=sk-ant-…
TARES_CATALOG=catalog.demo.yaml tares up

Either way, that one setup just created your entire AI SRE wiring; open the catalog file (or the use case’s Configuration tab) side by side with the console and find each piece:

• Three sources, all keyed by the same service label: demo_metrics (5xx rate, p99, DB connections, dependency health, polled from Prometheus), demo_logs (the api-server container’s request logs), and demo_alerts (the alerts Prometheus’s own rules fire).
• One view, service_timeline, joining the three on service. Open Explore, pick api-server: metrics, logs, and alerts merged on one clock. This single read is what every agent in this guide works from. Flip Agent view to see the exact text an agent gets.
• One trigger, incident, watching the view: it fires when an alert event arrives (sum(alert_active) > 0 over 1m, per service). The division of labour is deliberate and it is the one you already run: Prometheus keeps owning alerting; Tares ingests what fired, correlates it with the evidence, and handles the waking. Nothing is re-thresholded.
• One Tares agent, incident-first-look, subscribed to that trigger. Its whole definition is a prompt (read it under Tares agents): take the first look, tie the failure to specific evidence lines, suggest a next action.

Connect your MCP client

This is the pull half: you (or any agent you already use) investigating on demand. Start the MCP endpoint in a second shell and point your client at it:

tares mcp        # MCP endpoint on http://127.0.0.1:8788/mcp
 
# e.g. Claude Code:
claude mcp add --transport http tares http://localhost:8788/mcp

(Codex, Cursor, Claude Desktop, and others: per-client setup.) Then ask:

Use tares: what happened to api-server in the last 15 minutes?

The client calls query on service_timeline and answers from the correlated read. Right now the timeline is healthy traffic; the next step changes that.

Cause an incident and watch the SRE work

This is the push half. Break the api-server on purpose. On the use case page, pick a scenario and click Cause an incident; or from a shell:

curl -s -XPOST localhost:8080/demo/inject -H 'content-type: application/json' \
  -d '{"scenario": "error_spike"}'

Within ~30 seconds (Prometheus’s rules have a 15s for:, then Tares polls the alerts), the chain you built in step 2 runs on its own. Follow it in the console:

1. The 5xx rate climbs and HighErrorRate fires; the alert event lands in Explore next to the error logs and the metric that tripped it (the timeline turns red).
2. The incident trigger fires on the alert event. The firings are on the trigger’s page  and under Deliveries.
3. incident-first-look wakes, reads the correlated timeline, and writes a finding onto api-server’s timeline: what broke, since when, the likely cause tied to evidence lines, and a next action. Watch it appear under Runs on the use case page, in Explore, or under Tares agents → incident-first-look → Runs & findings.

That is the AI SRE working: nothing polled, nobody paged it, and the diagnosis is on the timeline before you’ve looked. Now ask your connected client the same question as before; it reads the evidence and the finding, so it starts from the previous conclusion instead of from scratch. That’s the loop closing: every investigation leaves the timeline smarter.

Try the other scenarios (latency, dependency_outage), and roll back with Clear the fault on the use case page or {"scenario": "clear"} from the shell; the alerts resolve and a resolved event lands in the timeline.

Point it at your real systems

Everything the demo wired is ordinary configuration, so this step repeats what step 2 did, against your systems instead. Four moves.

1. Recreate the three demo sources against your stack. In the console (Sources → Add source), or ask your connected agent to set them up. Keep everything keyed by the same service label:

2. Add the signal the demo lacks: deploys and config changes. Most real root causes are a change, so this is the highest-value source you’ll add. Create a webhook source (Sources → Add source → webhook) with a primary service label (read from the payload’s service field), event_type: deploy, and a text template like deploy {commit} by {author}: {message}. Creating it gives you a stable ingest URL. Then one line in your CD pipeline, run on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

Each POST lands as a deploy event on that service’s timeline, so the agent sees “deploy landed 14:02, errors started 14:04” the way you’d want to see it. A second webhook source with event_type: config covers config changes the same way.

3. Rebuild the view and triggers. A service_timeline view over your sources, keyed by service, and trigger conditions for the incident shapes you care about: keep reacting to fired alerts as the demo does, or threshold the typed fields directly (max(p99_ms) > 1000 for latency, an error-rate predicate for 5xx storms).

4. Pick who gets woken, per trigger.

• A Tares agent, like incident-first-look: edit the prompt for your stack, no infrastructure.
• Your own agent service, over a webhook: Tares POSTs the timeline to your endpoint and the investigation starts with zero reads. The delivery contract is three rules; the AI SRE cookbook  is a complete working implementation (an Anthropic SDK agent that queries over MCP, gets woken by the trigger, and writes back), including a measured comparison against a tool-per-system agent if you want the numbers.
• A Slack channel, so every firing and finding lands where your team already looks.