What does the incident trigger in the AI SRE demo fire on, and does Tares redo my alert thresholds?

The incident trigger in the Tares AI SRE demo fires when an alert event arrives on any of three sources keyed by service, and Tares does not redo your alert thresholds. The condition is sum(alert_active) > 0 over 1m, per service. Prometheus keeps owning alerting, and Tares ingests what fired, correlates it with the evidence and handles the waking.

How it works

The demo creates one trigger, incident. It watches three sources that all carry the same service label:

• demo_metrics: the 5xx rate, p99, database connections and dependency health, polled from Prometheus.
• demo_logs: the request logs of the api-server container.
• demo_alerts: the alerts that Prometheus’s own rules fire.

Because all three share service, Tares builds one timeline per service. The trigger evaluates per service, so an alert on api-server does not wake anything for another service.

The condition is sum(alert_active) > 0 over 1m. In practice the trigger fires when an alert event arrives. It does not look at the raw metric values and compare them to a number you set in Tares.

The thresholds live where they already did. Prometheus evaluates three alerting rules in the demo, HighErrorRate, HighLatency and DependencyDown, and each rule has its own for: window (15 seconds in the demo). Tares polls the alerts and receives the result. The split is deliberate:

• Prometheus decides what counts as an incident.
• Tares ingests what fired and correlates it with the metrics and logs on the same clock.
• Tares handles the waking, and nothing is re-thresholded.

The trigger wakes one Tares agent, incident-first-look. Its whole definition is a prompt: take the first look, tie the failure to specific evidence lines and suggest a next action. It reads the correlated timeline and writes a finding back onto the service’s timeline.

Set it up

You need Docker running, uv or pipx, and an Anthropic API key so the built-in agent can run. Full walkthrough: Build an AI SRE.

1. Start the demo system, which is an api-server, Prometheus with the three rules, and a traffic generator:

curl -O https://raw.githubusercontent.com/glassflow/tares/main/demo/docker-compose.yml
docker compose up -d

2. Install Tares and start it:

uv tool install tares          # or: brew install glassflow/tap/tares
tares up

3. Open http://127.0.0.1:8787, go to Projects and pick the AI SRE demo template. Paste your Anthropic key on the setup page and click Start. The three sources, the incident trigger and the incident-first-look agent are created for you.
4. Cause an incident with the Cause an incident button on the project page, or from a shell:

curl -s -XPOST localhost:8080/demo/inject -H 'content-type: application/json' \
  -d '{"scenario": "error_spike"}'

5. Within about 30 seconds, watch the chain run. The 5xx rate climbs, HighErrorRate fires and the alert event lands in Explore next to the error logs. The incident trigger fires, and its firings show on the trigger’s page and under Firings. incident-first-look wakes and writes a finding.
6. Clear the fault with Clear the fault on the project page. The alerts resolve and a resolved event lands on the timeline.

To use your own systems, recreate the three sources against your stack and keep them keyed by service. For alerts, use prometheus_alerts or alertmanager, so your alerting stays where it is.

What you get

• A trigger whose firing has a plain cause: an alert event that Prometheus already decided to raise.
• No second set of thresholds to maintain. Change a rule in Prometheus and the trigger follows, because it reacts to the alert, not to a copy of the limit.
• One correlated timeline per service, with the alert next to the metric that tripped it and the logs around it.
• A woken agent that starts from evidence instead of from scratch, and a finding written back so the next investigation starts from the previous conclusion.
• A visible trail. Firings show on the trigger’s page, and Setup, Advanced setup, Activity shows the firing and the run it woke as one thread.

When something else fits

• If you want Tares to react to a typed field directly, such as max(p99_ms) > 1000 for latency or an error-rate predicate for 5xx storms, write that condition on your own trigger. It is your choice, and the demo does not need it.
• If you want an agent you run yourself, point the trigger at your own agent service over a webhook instead of incident-first-look, or send firings to a Slack channel.

Related