How do I tell a real incident from noise when I see a disk filling up?

Tares lets a cheap triage agent check your disk signals on a schedule, answer “normal” in one line most of the time, and escalate only when a host’s counts move against the previous window. The escalated host goes to a main agent that finds the cause. You get one root-cause note per real problem instead of a page for every blip.

How it works

A schedule trigger fires every interval, whether or not anything happened. It gives the triage agent a summary instead of raw lines. For each label you name, the summary shows the count per value in this window against the window before, largest change first, plus a few recent lines.

For a disk filling up, the useful question is which host’s count jumped, not what each line says. A host that was quiet and is now loud stands out. A host that is steady at the same level as the window before does not.

The triage preset applies a simple rule:

• It considers only values that grew at least 3x and by at least 50 events since the window before, or are new with 50 or more.
• It ignores routine noise.
• It judges among what is left.

The numbers are written in the prompt, so you tune them by editing it.

Every triage run ends one of two ways:

• no_op: nothing needs a closer look. The run succeeds with no finding, and the runs table shows “no finding” with a one-line reason.
• finding with verdict investigate: one entity looks off. The finding is recorded on that entity and names the numbers that moved.

A handoff then starts the main agent on that same entity. It receives the triage finding, not the logs, and fetches its own evidence. This is the pattern in Watch, then escalate, and it sits in front of the alert-driven agent described in AI SRE.

Set it up

1. Put the sources that carry your disk signals in one project, and label the field you want counted so the triage agent can see which value moved. Put both agents in the same project.

2. Add a schedule trigger. On the project’s page click + next to Triggers, then fires: on a schedule. Or add it in the catalog. The interval is at least 1m.

triggers:
  - name: watch-web
    sources: [web_logs]
    key_field: service
    condition:
      every: 10m                          # at least 1m
      summary_by: [service, status_code]  # labels to count; default: the trigger's key field

Swap in your own source and labels. For disk signals that means the label that names the machine, so the counts are per host.

3. Create a Tares agent on that trigger and seed its prompt from the Triage (on a schedule) preset. Pick the cheapest model on your provider that calls tools reliably, and set a budget so the cadence cannot run up a bill.

4. Create the main agent in the same project and seed its prompt from Root cause after a handoff. Set its trigger to No trigger: only a handoff starts it.

5. Open the triage agent, and under When it concludes choose Add a handoff. Pick investigate, the main agent and a cooldown of 30m. In the catalog the pair looks like this:

agents:
  - name: triage
    trigger: watch-web
    concludes: true
    verdicts:
      - { verdict: investigate, when: one entity looks like a problem and needs a closer look }
    prompt: ...             # from the Triage (on a schedule) preset
    handoffs:
      - { verdict: investigate, agent: rca, cooldown: 30m }   # at most two root-cause runs per hour per entity
  - name: rca               # no trigger: it runs only when handed off
    concludes: true
    verdicts:
      - { verdict: rca, when: you found what is failing and why }
      - { verdict: resolved, when: it was noise, or it is already over }
    prompt: ...             # from the Root cause after a handoff preset

6. Raise the daily run limit. At 10 minutes the triage agent runs 144 times a day, above the default of 50 runs per agent per day. Raise it under Settings → Agents, or with TARES_AGENT_DAILY_CAP.

Use the handoff or a trigger over findings to wake the main agent, not both, or it runs twice per flag.

What you get

• A one-line “normal” for most ticks, so a disk that is steady or briefly noisy costs a fraction of a cent and wakes nobody.
• An investigate finding on the one host whose counts moved against the previous window, with the numbers that moved.
• A root-cause note from the main agent on the same entity, ending with rca, or with resolved when it was noise or is already over.
• At most two root-cause runs per hour per entity, because of the 30m cooldown.
• The whole chain in one place. On the project’s Overview, under What the agents found, one card shows the headline and the next step. Click it to see the tick, the triage finding and the root cause in order.
• A cost you can predict. A small model reading a summary costs well under a cent a run, and the main agent runs only when triage flags something.

To stop the watch, disable the triage agent or pause its schedule trigger. With no new investigate findings, the main agent goes quiet too.

When something else fits

• If you already know the exact condition that means trouble, an alert-driven agent or a condition trigger fits better than a scheduled watch. See AI SRE.
• If a single hard threshold is all you need, a condition trigger on the source is simpler than two agents.
• If counts rarely move in your sources, the 3x and 50 event thresholds in the prompt may need lowering before triage flags anything.

Related