What does it cost to have an AI SRE watch for error spikes every ten minutes?

Tares keeps a ten-minute error-spike watch cheap by running a small triage agent on a schedule and waking the main agent only when triage flags something. A small model reading a summary costs well under a cent a run. Ten minutes means 144 triage runs a day, above the default limit of 50 runs per agent per day, so raise the limit under Settings, Agents or with TARES_AGENT_DAILY_CAP.

How it works

The watch is two agents linked by a handoff, as described in Watch, then escalate. It sits in front of the alert-driven agent you build in Build an AI SRE.

• Schedule trigger. It fires every interval for all the events of its sources, whether or not anything happened. It hands the agent a summary of the window instead of raw lines: for each label you name, the count per value in this window against the window before, largest change first, plus a few recent lines. A stream of tens of thousands of lines becomes a page the model can read, which keeps each run small.
• Triage agent. It runs on every tick, so it is the part of the cost you pay on a fixed cadence. At 10 minutes that is 144 runs a day. Most runs end with no finding and a one-line reason.
• Main agent. It never wakes on its own. It runs only when triage concludes with the verdict investigate, and at most once per cooldown per entity. With a cooldown of 30 minutes, that is at most two root-cause runs per hour per entity.

So the steady cost is 144 small-model runs a day. The larger cost, the thorough root-cause run, appears only when something moves.

Set it up

1. Pick the sources. Put your request logs, for example a web_logs source labeled by service and status_code, and both agents in one project.
2. Add a schedule trigger. On the project’s page click + next to Triggers, then fires: on a schedule. Or add it in the catalog. The interval is at least 1m.

triggers:
  - name: watch-web
    sources: [web_logs]
    key_field: service
    condition:
      every: 10m                          # at least 1m
      summary_by: [service, status_code]  # labels to count; default: the trigger's key field

3. Add the triage agent. Create a Tares agent on watch-web and seed its prompt from the Triage (on a schedule) preset. Pick the cheapest model on your provider that calls tools reliably, and set a budget on the agent so the cadence cannot run up a bill.
4. Raise the daily limit. The default is 50 runs per agent per day, and a ten-minute schedule needs 144. Raise it under Settings, Agents, or set TARES_AGENT_DAILY_CAP. Leave it at 50 and the watch cannot complete a full day of ticks.
5. Add the handoff. Create the main agent from the Root cause after a handoff preset with no trigger, then add a handoff on the triage agent for the verdict investigate.

agents:
  - name: triage
    trigger: watch-web
    concludes: true
    verdicts:
      - { verdict: investigate, when: one entity looks like a problem and needs a closer look }
    prompt: ...             # from the Triage (on a schedule) preset
    handoffs:
      - { verdict: investigate, agent: rca, cooldown: 30m }   # at most two root-cause runs per hour per entity
  - name: rca               # no trigger: it runs only when handed off
    concludes: true
    verdicts:
      - { verdict: rca, when: you found what is failing and why }
      - { verdict: resolved, when: it was noise, or it is already over }
    prompt: ...             # from the Root cause after a handoff preset

6. Check the spend. Open the triage agent’s runs under Agents. Each row shows its outcome, its reason and its cost, so you see what a day of watching costs on your own model.

What you get

• A predictable floor: 144 triage runs a day, each well under a cent on a small model, so a full day of triage stays well under $1.44.
• A cap on the expensive part. The main agent runs only when triage flags an entity, and the cooldown limits it per entity.
• A budget on the triage agent, so the cadence cannot run up a bill.
• A cost shown on every run in the runs table, next to its outcome and reason, so you can tune the model, the interval or the thresholds with real numbers.
• A prompt you can tune. The preset considers only values that grew at least 3x and by at least 50 events since the window before, and ignores routine noise such as uptime probes. Edit the prompt to change those numbers.

To stop spending, disable the triage agent or pause its schedule trigger. With no new investigate findings, the main agent goes quiet too.

When something else fits

• If you only need to react to rules you already wrote, an alert-driven agent from Build an AI SRE fits, and it keeps running as before when you turn the watch off.
• If a longer interval is enough, widen every: a slower schedule means fewer runs a day, and at 50 runs a day or fewer you stay within the default limit.
• If you want to escalate only on a condition, such as several investigate findings in a window, use a trigger over findings instead of a handoff. Use one link or the other, not both, or the main agent runs twice per flag.

Related