How do I tell a real incident from noise when I see slow database queries?

Tares separates a real slow-query incident from noise with a cheap triage agent that compares counts to the previous window and escalates only when one entity jumps. Most runs end with a one-line “normal” and no finding. When a service’s slow-query count grows sharply against the window before, a main agent takes over that one entity for a root-cause analysis.

How it works

Two agents work in sequence, as described in Watch, then escalate. The pattern sits in front of the alert-driven agent from AI SRE.

• A schedule trigger fires every interval, whether or not anything happened. It does not pass raw log lines to the agent. It passes a summary of the window: for each label you name, the count per value in this window against the window before, largest change first, plus a few recent lines. Tens of thousands of slow-query lines become a page a model can read.
• The triage agent reads that summary. Its preset considers only values that grew at least 3x and by at least 50 events since the previous window, or are new with 50 or more. It ignores routine noise such as uptime probes and judges among what is left. The numbers live in the prompt, so you tune them by editing it.
• Every run ends one of two ways. With no_op, nothing needs a closer look, and the runs table shows “no finding” with a one-line reason. With a finding of verdict investigate, one entity looks like a problem, and the finding is recorded on it with the numbers that moved.
• The main agent is handed the triage finding, not the logs. It fetches its own evidence with read, stats and any MCP tools you give it, then writes its note on the same entity.

A steady level of slow queries produces no jump against the previous window, so triage stays quiet. A burst on one service produces a large change at the top of the summary, and that is what triage escalates.

Set it up

1. Put the source that carries your slow-query events in a project, and label the fields you want counted, for example service. Put both agents in the same project.
2. Add a schedule trigger on that source. Name the labels to count in summary_by. The interval is at least 1m.

triggers:
  - name: watch-web
    sources: [web_logs]
    key_field: service
    condition:
      every: 10m                          # at least 1m
      summary_by: [service, status_code]  # labels to count; default: the trigger's key field

3. Create a Tares agent on the trigger and seed its prompt from the Triage (on a schedule) preset. Pick the cheapest model on your provider that calls tools reliably, and set a budget. The preset ticks Ends every run with a conclusion and lists the investigate verdict.
4. Create the main agent in the same project from the Root cause after a handoff preset. Set its trigger to No trigger: only a handoff starts it.
5. On the triage agent, under When it concludes, choose Add a handoff. Pick investigate, the main agent and a cooldown of 30m. In the catalog the pair looks like this:

agents:
  - name: triage
    trigger: watch-web
    concludes: true
    verdicts:
      - { verdict: investigate, when: one entity looks like a problem and needs a closer look }
    prompt: ...             # from the Triage (on a schedule) preset
    handoffs:
      - { verdict: investigate, agent: rca, cooldown: 30m }   # at most two root-cause runs per hour per entity
  - name: rca               # no trigger: it runs only when handed off
    concludes: true
    verdicts:
      - { verdict: rca, when: you found what is failing and why }
      - { verdict: resolved, when: it was noise, or it is already over }
    prompt: ...             # from the Root cause after a handoff preset

6. Raise the daily run limit. At 10 minutes the triage agent runs 144 times a day, and the default limit is 50 runs per agent per day. Change it under Settings → Agents, or with TARES_AGENT_DAILY_CAP.

Use the handoff or a trigger over the findings source to wake the main agent, not both, or it runs twice per flag. The handoff is the simpler fit. The trigger is worth it when you need a condition, such as several investigate findings in a window or findings from more than one agent.

What you get

• Most ticks end as “no finding” rows with a reason and a cost, so you can see that the watch ran and judged the queries normal.
• A flagged service shows an investigate chip with a headline that names the numbers that moved.
• The main agent’s root cause appears on the project Overview under What the agents found, as one card for the whole chain. Open it to see the tick, the triage finding and the root cause in order.
• Cost stays low. A small model reading a summary costs well under a cent a run, and the main agent runs at most once per cooldown per entity.
• You can stop the watch at any time by disabling the triage agent or pausing its schedule trigger. The main agent then goes quiet too, and alert-driven agents keep running as before.

When something else fits

• If you already know the exact condition that means trouble, a condition trigger or an alert-driven agent reacts the moment it is true, without waiting for the next tick.
• If you need several findings in a window or findings from more than one agent before anyone wakes, use a trigger over the findings source instead of a handoff.

Related