How do I tell a real incident from noise when I see crash loops?

Tares lets a cheap triage agent compare crash counts against the previous window, answer “normal” in one line most of the time, and escalate only the entity whose counts moved. A root-cause agent then investigates that one entity. You see a short verdict per tick instead of a stream of restart lines.

How it works

A crash loop looks the same whether it is a real outage or a harmless blip: lines repeat. What separates them is movement. Tares puts that comparison in front of the model.

• A schedule trigger fires every interval, whether or not anything happened. Instead of raw lines, it hands the triage agent a summary: for each label you name, the count per value in this window against the window before, largest change first, plus a few recent lines.
• The triage agent reads that page. The preset prompt tells it to consider only values that grew at least 3x and by at least 50 events since the previous window, or are new with 50 or more. It ignores routine noise and judges what is left.
• Every run ends in one of two ways. no_op means nothing needs a closer look: the run succeeds with no finding and a one-line reason. A finding with verdict investigate means one entity looks off, and the finding names the numbers that moved.
• A handoff on the investigate verdict starts the main agent on the same entity. It receives the finding, not the logs, and fetches its own evidence with read, stats and any MCP tools you give it.

The triage agent is cheap and frequent. The main agent is thorough and rare. Each tick, verdict, handoff and root-cause note sits on the entity’s timeline. This pattern is the first level in front of the alert-driven agent described in AI SRE.

Set it up

1. Pick the source that carries your crash and restart lines, for example a docker_logs source that tails a container. Label the field you want counted, such as service, so the summary can show which value moved. Put both agents in one project.
2. Add a schedule trigger. On the project’s page click + next to Triggers, then fires: on a schedule. Or declare it in the catalog:

triggers:
  - name: watch-web
    sources: [web_logs]
    key_field: service
    condition:
      every: 10m                          # at least 1m
      summary_by: [service, status_code]  # labels to count; default: the trigger's key field

3. Point sources at your own source and set summary_by to the labels you want compared. If you leave it out, it defaults to the trigger’s key field.
4. Create the triage agent on that trigger and seed its prompt from the Triage (on a schedule) preset. Pick the cheapest model on your provider that calls tools reliably, and set a budget. If restarts are sparse in your fleet, edit the prompt: the 3x and 50-event numbers live there, so you can lower them.
5. Create the main agent in the same project, seed it from Root cause after a handoff, and choose No trigger: only a handoff starts it. Then open the triage agent and, under When it concludes, choose Add a handoff for investigate. In the catalog:

agents:
  - name: triage
    trigger: watch-web
    concludes: true
    verdicts:
      - { verdict: investigate, when: one entity looks like a problem and needs a closer look }
    prompt: ...             # from the Triage (on a schedule) preset
    handoffs:
      - { verdict: investigate, agent: rca, cooldown: 30m }   # at most two root-cause runs per hour per entity
  - name: rca               # no trigger: it runs only when handed off
    concludes: true
    verdicts:
      - { verdict: rca, when: you found what is failing and why }
      - { verdict: resolved, when: it was noise, or it is already over }
    prompt: ...             # from the Root cause after a handoff preset

6. Raise the daily run limit. At a 10 minute interval the triage agent runs 144 times a day, above the default of 50 per agent. Change it under Settings → Agents or with TARES_AGENT_DAILY_CAP.

Full walkthrough: Watch, then escalate.

What you get

• Most ticks end as “no finding” with a one-line reason, so a quiet fleet stays quiet.
• A loop that grows against the previous window is flagged on its own entity, for example service=checkout, with the numbers that moved.
• One root-cause run per flag, at most one per cooldown per entity. A cooldown of 30m means at most two per hour.
• One card under What the agents found on the project Overview, covering the tick, the triage finding and the root cause in order.
• Low cost: a small model reading a summary costs well under a cent a run, and the expensive agent runs only when triage flags something.
• An rca verdict when the main agent finds what is failing and why, or resolved when it was noise or is already over.

When something else fits

• If you already know the exact condition that means trouble, a condition trigger from Watch with triggers fits better than a periodic scan, because it fires only on that condition.
• If you need several investigate findings in a window, or findings from more than one agent, wake the main agent with a trigger over findings instead of a handoff. Use one link or the other, never both, or the main agent runs twice per flag.

Related