How do I escalate a disk filling up from a cheap triage agent to a root-cause agent?

Tares escalates a disk filling up with a handoff: a cheap triage agent ends its run with the verdict investigate, and Tares starts your root-cause agent on that one host. The handoff has its own cooldown, so the thorough agent runs only when triage says so, and at most once per cooldown for each entity. You wire it in the console or in the catalog, with no second trigger in between.

How it works

Two agents sit in one project, joined by a handoff.

• Triage agent. A schedule trigger wakes it on an interval. It reads a summary of the window, with counts per label value against the window before, and ends every run with the conclude tool. For most runs the answer is no_op and no finding is recorded.
• Root-cause agent. It has no trigger of its own. It runs only when a handoff reaches it, even if it is not switched on for any trigger.
• The handoff. It is three values: a verdict, the agent that takes over, and a cooldown. When a triage run ends with a finding whose verdict matches investigate (case does not matter), Tares starts the root-cause agent on the entity the finding is about, for example one host.

The root-cause agent receives the triage finding, not the raw data. It gathers further evidence itself with read, stats and any MCP tools you give it, then writes its conclusion on the same entity.

The cooldown allows at most one handoff from triage to that agent per entity inside the window. A handoff inside the cooldown starts no run, and the finishing run’s results say handoff to rca skipped: cooldown. If the target is already running for that host, or has hit its daily cap or budget, the handoff is recorded as a capped run with the reason. Chains stop at depth 3. See Hand off to another agent for every rule.

Set it up

1. Put a source that carries your disk usage in a project, and label the fields you want counted, such as the host and the mount point. Label the host as the primary label so each host is its own entity.
2. Add a schedule trigger over that source. Disks fill slowly, so a gentle interval is enough. Every 30 minutes is 48 runs a day, just under the default limit of 50 runs per agent per day.

triggers:
  - name: watch-disk
    sources: [node_metrics]
    key_field: host
    condition:
      every: 30m
      summary_by: [host, mountpoint]

3. Create the triage agent on watch-disk. Seed its prompt from the Triage (on a schedule) preset, pick the cheapest model that calls tools reliably, and set a budget. Edit the prompt so it says what a disk filling up means for you, because the preset’s thresholds are written in the prompt and you tune them there. The preset ticks Ends every run with a conclusion and lists the verdict investigate.
4. Create the root-cause agent in the same project. Seed its prompt from Root cause after a handoff, which comes with the verdicts rca and resolved, and set its trigger to No trigger: only a handoff starts it.
5. Open the triage agent, and under When it concludes choose Add a handoff. Pick investigate, pick the root-cause agent, and set the cooldown. Or write it in the catalog:

agents:
  - name: triage
    trigger: watch-disk
    concludes: true
    verdicts:
      - { verdict: investigate, when: one host has a disk that is filling up and needs a closer look }
    prompt: ...
    handoffs:
      - { verdict: investigate, agent: rca, cooldown: 30m }
  - name: rca
    concludes: true
    verdicts:
      - { verdict: rca, when: you found what is eating the disk and why }
      - { verdict: resolved, when: it was noise, or it is already over }
    prompt: ...

6. The cooldown defaults to 30m. A longer value suits a disk that stays full for hours, since it keeps the thorough agent from re-investigating the same host.

Use the handoff or a trigger over the findings source, not both, or the root-cause agent runs twice per flag. The full pattern is in Watch, then escalate.

What you get

• A frequent check that costs little. A small model reading a window summary costs well under a cent a run.
• A thorough analysis that runs only when triage says investigate, at most once per cooldown for each host.
• One thread per incident. On the project’s Activity page the tick, the triage finding and the root cause sit together, and the handed-off run is recorded with woken_by: handoff and the triage run as its parent.
• Nothing dropped silently. A skipped or capped handoff always leaves a recorded reason.
• A simple off switch. Disable the triage agent or pause its schedule trigger, and with no new investigate findings the root-cause agent goes quiet too.

This fits into the wider pattern described in AI SRE, where alert-driven agents keep running as before.

When something else fits

• Use a trigger over the findings source instead of a handoff when you need a condition, such as several investigate findings in a window or findings from more than one agent.
• Skip triage and run one agent straight from an alert trigger when an alert rule already says what you care about and the agent runs rarely.

Related