How do I escalate 5xx errors on an API from a cheap triage agent to a root-cause agent?

Tares escalates API 5xx errors with a handoff on the triage agent: when it concludes with the verdict investigate, the root-cause agent starts on that one service. The handoff has its own cooldown, so the thorough agent runs only when triage asks for it, and at most once per cooldown per entity. The root-cause agent needs no trigger of its own.

How it works

Two agents work in a chain, and a schedule trigger feeds the first one.

• A schedule trigger fires every interval over your API request logs. It gives the triage agent a digest, not raw lines. For each label you name, the digest shows the count per value in this window against the window before, largest change first, plus a few recent lines. If you label service and status_code, a jump in 502 or 503 responses for one service is visible at a glance.
• The triage agent is a small, cheap model. It must end every run with the conclude tool. Either nothing needs a look (a run with no finding) or it records a finding with the verdict investigate on one entity, such as service=checkout.
• The handoff is three values on the triage agent: the verdict, the agent that takes over and a cooldown. When a finding’s verdict matches (case does not matter), Tares starts the root-cause agent on the entity the finding is about and passes it the finding.
• The root-cause agent receives the finding, not the logs. It collects its own evidence with its tools, then writes its conclusion on the same entity.

The cooldown belongs to the handoff, not to the trigger. Triage can keep running every few minutes and keep writing findings, while the expensive agent starts at most once per entity inside the cooldown window. A handoff blocked by the cooldown starts no run, and the finishing run’s results say handoff to rca skipped: cooldown. If the target is already running for that entity, or has reached its daily cap or budget, the handoff is recorded as a capped run with the reason.

Set it up

1. Put your API request logs in a source, in the same project as both agents. Label the fields you want counted, such as service and status_code.
2. Add a schedule trigger over that source. On the project’s page click + next to Triggers, then fires: on a schedule. In the catalog it looks like this:

triggers:
  - name: watch-api
    sources: [api_logs]
    key_field: service
    condition:
      every: 10m                          # at least 1m
      summary_by: [service, status_code]  # labels to count

3. Create the triage agent on watch-api. Seed its prompt from the Triage (on a schedule) preset, pick the cheapest model that calls tools reliably and set a budget. The preset only considers values that grew at least 3x and by at least 50 events since the window before, so tune those numbers in the prompt if your API is quieter or busier. It also ticks Ends every run with a conclusion and lists the verdict investigate.
4. Create the root-cause agent in the same project. Seed its prompt from Root cause after a handoff and choose No trigger: only a handoff starts it. Tares runs a handed-off agent whether or not it is on for its own trigger.
5. Open the triage agent. Under When it concludes choose Add a handoff, pick investigate, pick the root-cause agent and set the cooldown. The default is 30m. In the catalog, the same wiring is:

agents:
  - name: api-triage
    trigger: watch-api
    concludes: true
    verdicts:
      - { verdict: investigate, when: one service is returning a burst of 5xx and needs a closer look }
    prompt: ...             # from the Triage (on a schedule) preset
    handoffs:
      - { verdict: investigate, agent: api-rca, cooldown: 30m }
  - name: api-rca           # no trigger: it runs only when handed off
    concludes: true
    verdicts:
      - { verdict: rca, when: you found what is failing and why }
      - { verdict: resolved, when: it was noise, or it is already over }
    prompt: ...             # from the Root cause after a handoff preset

6. Raise the daily run limit. Ten-minute ticks mean 144 triage runs a day, above the default of 50 runs per agent per day. Change it under Settings → Agents or with TARES_AGENT_DAILY_CAP.

The handoffs reference covers the depth limit, the cap on handoffs per agent and what the console shows when a verdict is never given. The Watch, then escalate guide walks through the full pattern.

What you get

• Most triage ticks end with a short no-finding line, so you pay a small model’s price for each check, which is well under a cent a run on a summary.
• When one service starts returning 5xx responses, the root-cause agent runs once on that service and writes its note on the same entity timeline.
• A 30 minute cooldown means at most two root-cause runs per hour per entity, however often triage flags it.
• A handed-off run is recorded with woken_by: handoff and the finishing run as its parent. The tick, the triage finding and the root cause read as one thread under Activity, and the Overview shows one card for the whole chain.
• Nothing is dropped silently. Cooldown skips and capped runs are recorded with their reason.

To stop the watch, disable the triage agent or pause its schedule trigger. With no new investigate findings, the root-cause agent goes quiet too. This sits in front of the alert-driven agent described in AI SRE.

When something else fits

• If you need a condition before escalating, such as several investigate findings in a window or findings from more than one agent, use a trigger over the findings source filtered to the triage agent. Use that or the handoff, not both, or the root-cause agent runs twice per flag.
• If a fixed alert rule already names the failure you care about, wake an agent from that rule directly and skip the triage layer.

Related