How do I tell a real incident from noise when I see high latency?

Tares lets a cheap triage agent check your sources every few minutes, say normal most of the time, and escalate only when counts move against the previous window. A main agent then runs a root-cause analysis on that one entity. The pattern is described in Watch, then escalate, and it sits in front of the alert-driven agent in AI SRE.

How it works

Two agents are linked by a handoff. The triage agent is cheap and frequent. The main agent is thorough and rare.

• A schedule trigger wakes triage. It fires every interval for all the events of its sources, whether or not anything happened.
• Triage reads a summary, not raw lines. For each label you name, it gets the count per value in this window against the window before, largest change first, plus a few recent lines. Tens of thousands of lines become a page the model can read.
• Triage applies a threshold. The Triage (on a schedule) preset tells the model to consider only values that grew at least 3x and by at least 50 events since the window before, or are new with 50 or more. It ignores routine noise such as uptime probes and judges among what is left. The numbers live in the prompt, so you tune them by editing it.
• Every run ends in one of two ways. With no_op, nothing needs a closer look, the run succeeds with no finding, and the runs table shows “no finding” with a one-line reason. With a finding and the verdict investigate, one entity looks off, and the finding names the numbers that moved.
• A handoff starts the main agent. On investigate, the main agent takes over on the same entity. It receives the triage finding, not the logs, and fetches its own evidence with read, stats and any MCP tools you give it. It then ends with rca or resolved, and resolved covers the case where it was noise or is already over.

Each tick, triage verdict, handoff and root-cause note is recorded on the entity’s timeline.

Set it up

1. Pick the sources. Put the sources that show your latency symptom, and both agents, in one project. Label the fields you want counted, such as service and status_code, so triage can see which values moved.
2. Add a schedule trigger. On the project’s page click + next to Triggers, then fires: on a schedule. Or add it in the catalog:

triggers:
  - name: watch-web
    sources: [web_logs]
    key_field: service
    condition:
      every: 10m                          # at least 1m
      summary_by: [service, status_code]  # labels to count; default: the trigger's key field

3. Add the triage agent. Create a Tares agent on watch-web and seed its prompt from the Triage (on a schedule) preset. Pick the cheapest model on your provider that calls tools reliably, and set a budget so the cadence cannot run up a bill.
4. Add the main agent and the handoff. Create the main agent in the same project, seed its prompt from Root cause after a handoff, and set its trigger to No trigger: only a handoff starts it. Then open the triage agent, choose Add a handoff under When it concludes, pick investigate, the main agent and cooldown 30m. In the catalog:

agents:
  - name: triage
    trigger: watch-web
    concludes: true
    verdicts:
      - { verdict: investigate, when: one entity looks like a problem and needs a closer look }
    prompt: ...             # from the Triage (on a schedule) preset
    handoffs:
      - { verdict: investigate, agent: rca, cooldown: 30m }   # at most two root-cause runs per hour per entity
  - name: rca               # no trigger: it runs only when handed off
    concludes: true
    verdicts:
      - { verdict: rca, when: you found what is failing and why }
      - { verdict: resolved, when: it was noise, or it is already over }
    prompt: ...             # from the Root cause after a handoff preset

5. Raise the daily run limit. At 10 minutes the triage agent runs 144 times a day, above the default limit of 50 runs per agent per day. Raise it under Settings → Agents, or with TARES_AGENT_DAILY_CAP.

If you need a condition before escalating, such as several investigate findings in a window, wake the main agent with a trigger over the findings source filtered to the triage agent’s findings instead of a handoff. Use one link or the other, not both, or the main agent runs twice per flag.

What you get

• Most ticks end as “no finding” with a one-line reason, so a quiet latency graph costs you a small model reading a summary, well under a cent a run.
• When something moves, you get one finding on one entity, for example service=checkout, with the numbers that changed against the window before.
• The main agent runs only when triage flags something, at most once per cooldown per entity, so root-cause work stays rare.
• The project Overview shows one card under What the agents found for the whole chain, with the headline and the next step. Click it to see the tick, the triage finding and the root cause in order.
• Under Setup → Advanced setup, the triage agent’s runs show mostly “no finding” rows with their reason and cost, and a flagged run shows an investigate chip.
• To stop the watch, disable the triage agent or pause its schedule trigger. The main agent then goes quiet too, and alert-driven agents keep running as before.

When something else fits

• If you already know the exact condition that matters, an alert-driven agent from AI SRE fits better. Watch-then-escalate is for what nobody thought to write a rule for.
• If a run every few minutes is more than you want to spend, lengthen the interval (the minimum is 1m) or pause the schedule trigger. Read the full steps in Watch, then escalate.

Related