How do I escalate crash loops from a cheap triage agent to a root-cause agent?

Tares escalates a crash loop with a handoff on the triage agent: when its run concludes with the verdict investigate, the root-cause agent starts on that one entity. The handoff carries its own cooldown, so the thorough agent runs only when triage asks for it, and no more than once per cooldown for the same entity. The root-cause agent needs no trigger of its own.

How it works

A handoff is three values: a verdict, the agent that takes over, and a cooldown. Both agents live in one project, and the handoff is part of that project’s wiring.

• The triage agent wakes on a schedule trigger. Each tick it gets a summary of the window: for each label you name, the count per value against the window before, largest change first, plus a few recent lines.
• Triage ends every run with the conclude tool. It either finds nothing (a run with no finding and a one-line reason) or records a finding with the verdict investigate on the entity that looks wrong, for example service=checkout.
• When the verdict matches, Tares starts the root-cause agent on that entity and hands it the triage finding, not the raw logs. The root-cause agent pulls its own evidence with read, stats and any MCP tools you give it.
• The cooldown limits handoffs from triage to that agent to one per entity per window. A handoff inside the window starts no run, and the finishing run’s results say handoff to rca skipped: cooldown.

This is the pattern from Watch, then escalate, applied to restarts. The full rules are on the Handoffs page. The alert-driven agent this sits in front of is described in AI SRE.

Set it up

1. Put the sources that show your restarts in one project and label the fields you want counted, such as service.
2. Add a schedule trigger. It fires every interval whether or not anything happened, and every is at least 1m.
3. Create the triage agent on that trigger. Use the Triage (on a schedule) preset, pick the cheapest model on your provider that calls tools reliably, and set a budget. Edit the prompt so that repeated restarts of one service are what counts as a problem and routine noise is ignored.
4. Tick Ends every run with a conclusion and list investigate under Verdicts it can give.
5. Create the root-cause agent in the same project. Use the Root cause after a handoff preset and choose No trigger: only a handoff starts it.
6. Open the triage agent, and under When it concludes choose Add a handoff. Pick investigate, pick the root-cause agent and set the cooldown. The default is 30m.

In the catalog the same wiring reads:

triggers:
  - name: watch-web
    sources: [web_logs]
    key_field: service
    condition:
      every: 10m                          # at least 1m
      summary_by: [service, status_code]  # labels to count; default: the trigger's key field
 
agents:
  - name: triage
    trigger: watch-web
    concludes: true
    verdicts:
      - { verdict: investigate, when: one entity looks like a problem and needs a closer look }
    prompt: ...             # from the Triage (on a schedule) preset
    handoffs:
      - { verdict: investigate, agent: rca, cooldown: 30m }   # at most two root-cause runs per hour per entity
  - name: rca               # no trigger: it runs only when handed off
    concludes: true
    verdicts:
      - { verdict: rca, when: you found what is failing and why }
      - { verdict: resolved, when: it was noise, or it is already over }
    prompt: ...             # from the Root cause after a handoff preset

A 10 minute schedule is 144 triage runs a day, above the default limit of 50 runs per agent per day. Raise it under Settings → Agents or with TARES_AGENT_DAILY_CAP.

Use either the handoff or a trigger over the findings source to wake the root-cause agent, not both. With both, it runs twice for every flag.

What you get

• A cheap first level that reads a summary each tick and usually ends with a short “no finding” line.
• A thorough second level that starts only on an investigate verdict, once per entity per cooldown. With 30m that is at most two root-cause runs an hour for a looping service, however often triage flags it again.
• One thread per incident. The handed-off run is recorded with woken_by: handoff and the finishing run as its parent, so the tick, the triage finding and the root cause read in order under the project’s Activity.
• One card on the project Overview under What the agents found, with the headline and the next step.
• Visible limits. If the root-cause agent is already running for that entity, or has hit its daily cap or budget, the handoff is recorded as a capped run with the reason. Chains stop at depth 3, and the next handoff is recorded as capped with handoff chain stopped at depth 3.
• A simple off switch. Disable the triage agent or pause its schedule trigger, and the root-cause agent goes quiet with it.

When something else fits

• Use a trigger over the findings source when you need a condition before escalating, such as several investigate findings in a window or findings from more than one agent.
• For alerts you already know how to write, a normal alert-driven agent as in AI SRE needs no triage layer.

Related