What does it cost to have an AI SRE watch for failed deploys every ten minutes?
Tares keeps the cost of a ten-minute failed-deploy watch low because a small triage model reads a summary and costs well under a cent a run. The main agent runs only when triage flags something. Ten minutes is 144 runs a day, above the default limit of 50 runs per agent per day, so raise the limit under Settings, Agents or with TARES_AGENT_DAILY_CAP, and give the triage agent a budget.
How it works
The watch is two agents linked by a handoff, as described in Watch, then escalate. It is one pattern in the alert-driven setup from AI SRE.
- Triage agent. A schedule trigger fires every interval. At ten minutes that is 144 runs a day. Each run hands the agent a summary of the window instead of raw lines: for each label you name, the count per value against the window before, largest change first, plus a few recent lines. A busy stream of tens of thousands of lines becomes one page the model can read, which keeps each run small.
- Main agent. It has no trigger of its own. It starts only when triage ends a run with the verdict
investigate, and at most once per cooldown per entity. With a30mcooldown that is at most two root-cause runs per hour per entity.
Most triage runs end as no_op: the run succeeds with no finding and a one-line reason. Those are the cheap runs, and they make up almost all of the 144 a day.
The cost has three parts:
- Triage runs. 144 runs a day at well under a cent each comes to well under $1.44 a day.
- Main-agent runs. These happen only when triage flags an entity, and the cooldown caps them.
- The agent budget. It stops the cadence from running up a bill.
Set it up
- Label the fields you want counted on the source that carries your deploy events, so the triage agent can see which values moved. Put both agents in one project.
- Add a schedule trigger. On the project’s page click + next to Triggers, then fires: on a schedule. Or add it in the catalog. The example below is the one from the guide, so swap in your own source and labels:
triggers:
- name: watch-web
sources: [web_logs]
key_field: service
condition:
every: 10m # at least 1m
summary_by: [service, status_code] # labels to count; default: the trigger's key field- Create the triage agent on that trigger and seed its prompt from the Triage (on a schedule) preset. Pick the cheapest model on your provider that calls tools reliably.
- Set a budget on the triage agent so the cadence cannot run up a bill.
- Raise the daily run limit, because 144 runs a day is above the default of 50 per agent. Change it under Settings, Agents, or set
TARES_AGENT_DAILY_CAP. - Create the main agent with no trigger, seed it from Root cause after a handoff, then add a handoff on the triage agent: verdict
investigate, the main agent, cooldown30m. - Watch the spend. Open the triage agent’s runs under Agents. Each row shows its outcome, its reason and its cost, so you can check the real figure against your budget.
What you get
- A predictable bill: 144 small triage runs a day, each well under a cent, plus main-agent runs only when something is flagged.
- A limit that fits the schedule, so the watch is not held to 50 runs a day.
- A budget on the triage agent as a ceiling on spend.
- A cost shown on every run, with most rows reading “no finding” and a reason.
- A full record on the entity’s timeline: each tick, each triage verdict, each handoff and each root-cause note.
When something else fits
- If a ten-minute cadence is more than you need, lengthen the
everyinterval. The minimum is1m, and fewer runs a day stay under the default limit sooner. - If you only want to react to alerts you already have, an alert-driven agent runs only when an alert arrives, with no schedule.
- To stop the watch, disable the triage agent or pause its schedule trigger. The main agent then goes quiet too.
Related
Last updated on