How do I escalate error spikes from a cheap triage agent to a root-cause agent?

In Tares, you escalate error spikes with a handoff on the triage agent. When triage concludes with the verdict investigate, Tares starts your root-cause agent on that one entity, at most once per cooldown. The root-cause agent runs only when triage says investigate, so it stays thorough and rare while triage stays cheap and frequent.

How it works

Two agents are linked by a handoff, which is three values: a verdict, the agent that takes over, and a cooldown.

• A schedule trigger wakes the triage agent every interval. It hands the agent a summary of the window instead of raw lines: for each label you name, such as service and status_code, the count per value against the window before, largest change first, plus a few recent lines.
• Triage ends every run with the conclude tool. Either nothing needs a closer look, which is a success with no finding, or one entity looks off and the finding carries the verdict investigate.
• When the verdict matches the handoff (case does not matter), Tares starts the root-cause agent on the entity the finding is about. It receives the finding, not the logs, and fetches its own evidence with read, stats and any MCP tools you give it.
• The cooldown is per handoff and per entity. At most one handoff from triage to the root-cause agent starts for that entity within the cooldown. A handoff inside it starts no run, and the finishing run’s results say handoff to rca skipped: cooldown.

The root-cause agent has no trigger of its own, so it never wakes except by handoff. Each handed-off run is recorded with woken_by: handoff and the finishing run as its parent, so the triage finding and the root cause read as one thread on the project’s Activity.

The full rules are on Hand off to another agent.

Set it up

1. Put your sources and both agents in one project. Label the fields you want counted, such as service and status_code, so triage can see which values moved.
2. Add a schedule trigger. On the project’s page click + next to Triggers, then fires: on a schedule. In the catalog it looks like this:

triggers:
  - name: watch-web
    sources: [web_logs]
    key_field: service
    condition:
      every: 10m                          # at least 1m
      summary_by: [service, status_code]  # labels to count; default: the trigger's key field

3. Create the triage agent on watch-web and seed its prompt from the Triage (on a schedule) preset. Pick the cheapest model on your provider that calls tools reliably, and set a budget. The preset ticks Ends every run with a conclusion and lists the verdict investigate. Its thresholds, 3x growth and at least 50 events, are in the prompt, so edit it to tune them for your error spikes.
4. Create the root-cause agent in the same project and seed its prompt from Root cause after a handoff. Set its trigger to No trigger: only a handoff starts it.
5. Open the triage agent, and under When it concludes choose Add a handoff. Pick investigate from the verdict dropdown, pick the root-cause agent, and keep the cooldown at 30m. In the catalog the pair looks like this:

agents:
  - name: triage
    trigger: watch-web
    concludes: true
    verdicts:
      - { verdict: investigate, when: one entity looks like a problem and needs a closer look }
    prompt: ...             # from the Triage (on a schedule) preset
    handoffs:
      - { verdict: investigate, agent: rca, cooldown: 30m }   # at most two root-cause runs per hour per entity
  - name: rca               # no trigger: it runs only when handed off
    concludes: true
    verdicts:
      - { verdict: rca, when: you found what is failing and why }
      - { verdict: resolved, when: it was noise, or it is already over }
    prompt: ...             # from the Root cause after a handoff preset

6. Raise the daily run limit. A 10 minute schedule is 144 triage runs a day, above the default of 50 per agent per day. Change it under Settings → Agents, or with TARES_AGENT_DAILY_CAP.

The walkthrough with every step is in Watch, then escalate.

What you get

• A root-cause agent that runs only when triage says investigate, and at most once per cooldown per entity. With 30m that is at most two root-cause runs per hour per entity.
• A cheap watch. A small model reading a summary costs well under a cent a run.
• One thread per incident. The Overview shows one card under What the agents found, and clicking it shows the tick, the triage finding and the root cause in order.
• No silent drops. If the root-cause agent is already running for that entity, or has hit its daily cap or budget, the handoff is recorded as a capped run with the reason. A chain that goes past three handoffs in a row is recorded as capped with handoff chain stopped at depth 3.
• A simple off switch. Disable the triage agent or pause its schedule trigger, and with no new investigate findings the root-cause agent goes quiet too.

When something else fits

A handoff is the simpler link, and it shows on the triage agent’s page under When it concludes. Use a trigger over the findings source instead when you need a condition, such as several investigate findings in a window or findings from more than one agent. That trigger must be filtered to another agent’s findings, because a trigger over all findings or over the agent’s own is refused:

triggers:
  - name: escalate
    sources: [findings]
    key_field: service
    filters:
      - {field: agent, op: eq, value: triage}
      - {field: verdict, op: eq, value: investigate}
    condition: {aggregate: count, predicate: "> 0", window: 5m}
    cooldown: 30m    # at most two root-cause runs per hour per entity

Use one link or the other, not both, or the root-cause agent runs twice per flag. If you already have alert rules for the errors you expect, keep them. The triage agent covers what nobody thought to write a rule for, and alert-driven agents keep running as before.

Related