How do I tell a real incident from noise when I see error spikes?

Tares tells a real incident from noise by comparing each label’s counts in the current window against the window before it. A cheap triage agent reads that comparison on a schedule and says normal most of the time. It escalates one entity to a root-cause agent only when the counts move against the previous window.

How it works

You split the work between two agents, so the expensive one runs rarely.

• A schedule trigger fires every interval, whether or not anything happened. Instead of raw lines, it gives the agent a summary: for each label you name, the count per value in this window against the previous window, largest change first, plus a few recent lines. Tens of thousands of log lines become one page a small model can read.
• The triage agent applies a rule of thumb from its preset. It considers only values that grew at least 3x and by at least 50 events since the previous window, or that are new with 50 or more events. It ignores routine noise such as uptime probes, then judges among what is left.
• Every run ends with a conclusion. If nothing needs a closer look, the run ends as no_op. It counts as a success with no finding, and the runs table shows “no finding” with a one-line reason. If one entity looks off, the run records a finding with the verdict investigate, on that entity (for example service=checkout), naming the numbers that moved.
• The main agent takes over only on that verdict. It receives the triage finding, not the logs, and fetches its own evidence with read, stats and any MCP tools you give it. It then writes its root-cause note on the same entity.

The 3x and 50-event numbers live in the prompt, so you tune them by editing it. A spike that stays under them, or that hits only probe traffic, ends as “normal” and never reaches the main agent.

Set it up

1. Label the fields you want compared. Put your sources and both agents in one project. Give the sources labels such as service and status_code, so the triage agent can see which values moved. For example, request logs in a web_logs source labeled by service.
2. Add a schedule trigger. On the project’s page click + next to Triggers, then fires: on a schedule. Or add it in the catalog:

triggers:
  - name: watch-web
    sources: [web_logs]
    key_field: service
    condition:
      every: 10m                          # at least 1m
      summary_by: [service, status_code]  # labels to count; default: the trigger's key field

3. Create the triage agent on watch-web. Seed its prompt from the Triage (on a schedule) preset. Pick the cheapest model on your provider that calls tools reliably, and set a budget so the cadence cannot run up a bill. The preset ticks Ends every run with a conclusion and lists investigate as the verdict it can give.
4. Create the main agent and link it with a handoff. Seed its prompt from Root cause after a handoff and set its trigger to No trigger: only a handoff starts it. On the triage agent, under When it concludes, choose Add a handoff, pick investigate, pick the main agent and set a cooldown of 30m. In the catalog:

agents:
  - name: triage
    trigger: watch-web
    concludes: true
    verdicts:
      - { verdict: investigate, when: one entity looks like a problem and needs a closer look }
    prompt: ...             # from the Triage (on a schedule) preset
    handoffs:
      - { verdict: investigate, agent: rca, cooldown: 30m }   # at most two root-cause runs per hour per entity
  - name: rca               # no trigger: it runs only when handed off
    concludes: true
    verdicts:
      - { verdict: rca, when: you found what is failing and why }
      - { verdict: resolved, when: it was noise, or it is already over }
    prompt: ...             # from the Root cause after a handoff preset

5. Raise the daily run limit. At 10 minutes the triage agent runs 144 times a day, above the default limit of 50 runs per agent per day. Raise it under Settings → Agents, or with TARES_AGENT_DAILY_CAP.

The full walkthrough, including the alternative of waking the main agent with a trigger over findings, is in Watch, then escalate.

What you get

• Quiet by default. Most runs end as “no finding” with a one-line reason, so you read nothing when nothing moved.
• One flag per real problem. An investigate finding names a single entity and the numbers that moved, instead of a flood of alerts.
• A root cause only when it is warranted. The main agent runs after a handoff, at most once per cooldown per entity. With 30m, that is at most two root-cause runs per hour per entity.
• A visible trail. The project Overview shows one card under What the agents found for the whole chain. Click it to see the tick, the triage finding and the root cause in order. In Setup → Advanced setup, the triage agent’s runs show the outcome, reason and cost of each tick.
• A low bill. A small model reading a summary costs well under a cent a run.
• An easy off switch. Disable the triage agent or pause its schedule trigger, and the main agent goes quiet too. Alert-driven agents keep running as before.

When something else fits

This pattern compares counts between windows, so it suits spikes you can see in labeled volume. If you already know the exact condition that matters, a condition trigger that fires on a count or threshold is the more direct tool, and your root-cause agent can wake from it straight away. The triage layer earns its place when you want coverage for things nobody wrote a rule for.

If you need several investigate findings in a window, or findings from more than one agent, before the main agent wakes, use a trigger over findings instead of a handoff. Use one link or the other, not both, or the main agent runs twice per flag. The same guide covers both routes.

Related