How do I simulate a latency problem or a dependency outage in the demo, and how do I roll the fault back?

Tares lets you cause a latency problem or a dependency outage in the demo by choosing the latency or dependency_outage scenario, alongside error_spike. You post the scenario to the demo inject endpoint or pick it on the project page and click Cause an incident. To roll back, click Clear the fault on the project page or post the clear scenario, and the alerts resolve and a resolved event lands in the timeline.

How it works

The demo stack in the Build an AI SRE guide includes a small api-server with a fault switch. Prometheus scrapes it and evaluates three alerting rules: HighErrorRate, HighLatency and DependencyDown. Tares polls the alerts those rules fire and correlates them with metrics and logs on one timeline per service.

The demo has these scenarios:

• error_spike breaks the api-server with a burst of 5xx errors.
• latency makes the api-server slow.
• dependency_outage takes down a dependency of the api-server.
• clear rolls the fault back.

You choose a scenario in one of two ways. On the project page, pick a scenario and click Cause an incident. Or post the scenario name to the demo inject endpoint on the api-server, port 8080.

The chain then runs on its own. Prometheus’s rules have a 15 second for:, then Tares polls the alerts, so expect the first signs within about 30 seconds. The incident trigger fires on the alert event. The incident-first-look agent wakes, reads the correlated timeline and writes a finding onto the api-server timeline.

Set it up

1. Start the demo system. This is the compose file from the guide:

   curl -O https://raw.githubusercontent.com/glassflow/tares/main/demo/docker-compose.yml
   docker compose up -d

2. Install Tares and start it with tares up. Open the console at http://127.0.0.1:8787, go to Projects, pick the AI SRE demo template and click Start. The built-in agent needs an Anthropic key, which you paste on the same page or add under Settings, Model providers. The project page now has the Cause an incident button.

3. Cause a latency problem or a dependency outage. In the console, pick latency or dependency_outage on the project page and click Cause an incident. From a shell, post the scenario to the inject endpoint. This example uses the latency scenario. For an outage, put dependency_outage in the scenario value:

   curl -s -XPOST localhost:8080/demo/inject -H 'content-type: application/json' \
     -d '{"scenario": "latency"}'

4. Watch it play out. Open Explore and pick api-server. The alert event lands next to the logs and the metric that tripped it. The firings show on the incident trigger’s page and under Firings, and the finding appears on the project’s Overview under What the agents found.

5. Roll the fault back. Click Clear the fault on the project page, or post the clear scenario:

   curl -s -XPOST localhost:8080/demo/inject -H 'content-type: application/json' -d '{"scenario": "clear"}'

6. Confirm the rollback. The alerts resolve and a resolved event lands in the timeline for api-server.

What you get

• Three incident shapes to practice on, each with its own evidence: error_spike, latency and dependency_outage.
• One correlated timeline per service, where metrics, logs and the fired alerts sit on one clock.
• An automatic response. The incident trigger fires and incident-first-look writes a finding that names what broke, since when, the likely cause tied to evidence lines, and a next action.
• A clean rollback. Clearing the fault resolves the alerts and puts a resolved event on the timeline, so you can see the whole incident from start to end.
• A repeatable loop. If you ask a connected MCP client about the api-server after an incident, it reads the evidence and the finding, so it starts from the previous conclusion.

When something else fits

• The scenarios are demo faults on the bundled api-server. To watch your own services, follow the last step of the Build an AI SRE guide, which recreates the sources, triggers and agents against your real systems.
• When you finish, post the clear scenario before you run docker compose down, so the demo is not left in a faulted state.

Related