Skip to Content
Tares

I caused an incident in the demo but no finding appeared, where do I look?

Tares shows where a demo incident stalled in four places: Explore, the incident trigger’s firings, Activity, and the Runs of incident-first-look. Wait about 30 seconds after you cause the incident, because Prometheus rules have a 15s for and then Tares polls the alerts. Then follow the chain in order. The built-in agent also needs an Anthropic API key to run.

How it works

The demo wires one chain, and each link leaves a trace you can check. This is the same chain the AI SRE guide builds.

  • The api-server fault makes the 5xx rate climb, and the Prometheus rule HighErrorRate fires after its 15s for.
  • The demo_alerts source polls Prometheus for fired alerts, so the alert event lands on the api-server timeline.
  • The incident trigger watches the three demo sources, all keyed by the service label. It fires when an alert event arrives (sum(alert_active) > 0 over 1m, per service).
  • The incident trigger wakes the Tares agent incident-first-look. The agent reads the correlated timeline and writes a finding onto it.

If a link is missing, everything after it is missing too. Work from the start and stop at the first link that did not happen. That link is where your problem is.

Set it up

  1. Wait about 30 seconds. The 15s for on the Prometheus rule comes first, then Tares polls the alerts. A finding right after you click Cause an incident is too early.

  2. Check Explore. Open Explore and pick api-server. You should see the HighErrorRate alert event next to the error logs and the metric that tripped it, and the timeline turns red. If the alert is not there, the incident never reached Tares. Check that the demo stack is still running in Docker and that the fault was injected, using the Cause an incident button on the project page. Nothing downstream can fire without this alert event.

  3. Check that the incident trigger fired. Open the trigger’s page at http://127.0.0.1:8787/triggers/incident, or open Firings in the console. A firing there means the alert reached the trigger and the trigger woke the agent. No firing means the problem is between the alert and the trigger.

  4. Check Activity and Runs. Open Setup → Advanced setup. Activity shows the firing and the run it woke as one thread. Then open Agents → incident-first-look → Runs to see every run. A firing with no run, or a run that did not finish, points at the agent.

  5. Check the Anthropic key. The built-in agent needs an Anthropic API key to run. Paste one on the AI SRE demo setup page, or add it under Settings, Model providers. If you started from the demo catalog, export the key before you start Tares:

export ANTHROPIC_API_KEY=sk-ant-…
TARES_CATALOG=catalog.demo.yaml tares up
  1. Look in the right place for the finding. A finding appears on the project’s Overview under What the agents found, with a headline and the next step. It is also on api-server’s timeline in Explore.

The catalog only seeds an empty instance. If you ran tares up earlier in the same data directory, start on a fresh one with --data-dir ~/tares-sre.

What you get

  • You know which link broke: the alert, the trigger firing, the run, or the finding.
  • You see the firing and the run it woke as one thread under Activity, so you do not have to match timestamps across pages.
  • You can open every run of incident-first-look under Runs and read what it did.
  • Once the chain works, the finding shows up on the project Overview and on the api-server timeline. A connected MCP client then starts from that finding instead of from scratch.

When something else fits

  • If HighErrorRate never shows in Explore, the fault or the Prometheus side is the problem, not the agent. Start with the demo stack and the Cause an incident button.
  • If you want to look at the evidence yourself rather than wait for a finding, connect an MCP client with tares mcp and ask it what happened to api-server. It reads the same correlated timeline.

Still stuck? Get help

Last updated on
Quickstart