How do I tell a real incident from noise when I see failed deploys?

Tares tells a real incident from deploy noise by comparing counts. A cheap triage agent reads a summary of your deploy events every few minutes and answers normal most of the time. It escalates one entity to a root-cause agent only when its failed-deploy counts move against the previous window. A single red build never pages anyone, and a service that suddenly fails far more than before does.

How it works

This is the watch, then escalate pattern from Watch, then escalate, built in front of the alert-driven agent described in AI SRE. Two agents work in sequence.

• A schedule trigger fires every interval for all the events of the sources you name, whether or not anything happened. It does not pass raw lines to the agent. It passes a summary: for each label you name, the count per value in this window against the window before, largest change first, plus a few recent lines.
• The triage agent reads that summary with a cheap model. Its preset tells it to consider only values that grew at least 3x and by at least 50 events since the window before, or are new with 50 or more. It ignores routine noise and judges among what is left.
• The main agent is handed the triage finding, not the logs. It fetches its own evidence with read, stats and any MCP tools you give it, then writes a root-cause note on the same entity.

Every run ends one of two ways. With no_op, nothing needs a closer look: the run succeeds with no finding and a one-line reason. With a finding whose verdict is investigate, one entity looks off, and the finding names the numbers that moved. For failed deploys, that means a service whose failures jumped against the last window gets flagged, while a service with its usual trickle of failures stays at normal.

The 50-event floor is written in the prompt. Deploys are usually fewer events than request logs, so lower the floor by editing the prompt to match your deploy volume.

Set it up

1. Put your deploy events in a source and label the fields you want counted, such as service and a status label, so the triage agent can see which values moved. Put both agents in one project.
2. Add a schedule trigger. On the project’s page click + next to Triggers, then fires: on a schedule. Or write it in the catalog:

triggers:
  - name: watch-deploys
    sources: [deploys]
    key_field: service
    condition:
      every: 10m                       # at least 1m
      summary_by: [service, status]    # labels to count

3. Create a Tares agent on watch-deploys and seed its prompt from the Triage (on a schedule) preset. Pick the cheapest model on your provider that calls tools reliably, and set a budget.
4. Create the main agent in the same project from the Root cause after a handoff preset. Set its trigger to No trigger: only a handoff starts it.
5. On the triage agent, under When it concludes, choose Add a handoff. Pick investigate, the main agent and a cooldown of 30m. In the catalog:

agents:
  - name: triage
    trigger: watch-deploys
    concludes: true
    verdicts:
      - { verdict: investigate, when: one entity looks like a problem and needs a closer look }
    prompt: ...             # from the Triage (on a schedule) preset
    handoffs:
      - { verdict: investigate, agent: rca, cooldown: 30m }
  - name: rca               # no trigger: it runs only when handed off
    concludes: true
    verdicts:
      - { verdict: rca, when: you found what is failing and why }
      - { verdict: resolved, when: it was noise, or it is already over }
    prompt: ...             # from the Root cause after a handoff preset

6. Raise the daily run limit. At a 10 minute interval the triage agent runs 144 times a day, above the default of 50 runs per agent per day. Raise it under Settings → Agents or with TARES_AGENT_DAILY_CAP.

Use a handoff or a trigger over findings to wake the main agent, not both, or it runs twice per flag.

What you get

• Most ticks end with a no_op row that reads “no finding” and a one-line reason, so a quiet day of deploys costs little. A small model reading a summary costs well under a cent a run.
• When failed deploys move against the previous window, you get one finding on the affected entity, such as service=checkout, with the numbers that changed.
• The main agent runs only after that flag, at most once per cooldown per entity. With a 30m cooldown that is at most two root-cause runs per hour.
• The project Overview shows one card under What the agents found for the whole chain, with the headline and the next step. Click it to see the tick, the triage finding and the root cause in order.
• Triage runs sit in the agent’s runs table, each with its outcome, reason and cost, so you can check that the threshold suits your deploy volume.
• To stop the watch, disable the triage agent or pause its schedule trigger. The main agent goes quiet too.

When something else fits

• If you already know the exact failure you care about, such as any failed deploy of one service, an alert-driven agent in AI SRE fits better than a count comparison.
• If you want escalation only after several investigate findings in a window, or findings from more than one agent, wake the main agent with a trigger over findings instead of a handoff. Watch, then escalate shows how.

Related