Can an AI SRE check for 5xx errors on an API on a schedule instead of waiting for an alert?

Yes, Tares can check your API for 5xx errors on a fixed interval, with no alert rule involved. A schedule trigger fires every interval and gives a triage agent a summary of the window. When that agent finds one service with a jump in error codes, it hands that service to the main agent for a root-cause note. Most ticks end with a one-line “normal”.

How it works

Three pieces work together, all in one project:

• Sources. Your API’s request logs, with the fields you want counted turned into labels, such as service and status_code.
• A schedule trigger. It fires every interval, whether or not anything happened. It does not pass raw log lines to the agent. It passes a summary: for each label you name, the count per value in this window against the window before, largest change first, plus a few recent lines. If your API writes tens of thousands of lines in ten minutes, the agent still reads one short page.
• Two agents. The triage agent is cheap and runs often. The main agent is thorough and runs rarely. They are linked by a handoff.

For 5xx checks, the status_code label is what makes a jump in 502 or 503 responses visible next to the service that returned them. The triage preset considers only values that grew at least 3x and by at least 50 events since the window before, or are new with 50 or more. It ignores routine noise such as uptime probes. The numbers live in the prompt, so you tune them by editing it.

Every triage run ends one of two ways. With no_op, nothing needs a closer look, and the run is a success with no finding. With a finding carrying the verdict investigate, one entity looks off, and the finding names the numbers that moved. That verdict triggers the handoff. The main agent receives the triage finding, not the logs, and fetches its own evidence with read, stats and any MCP tools you give it. This is the pattern in Watch, then escalate, and it sits in front of the alert-driven agent from AI SRE.

Set it up

1. Label your API logs. Put both agents in one project. In the source that holds your API’s request logs, add labels for service and status_code.
2. Add the schedule trigger. On the project’s page click + next to Triggers, then fires: on a schedule. Or declare it in the catalog. The interval is at least 1m, and summary_by lists the labels to count.
3. Create the triage agent. Point it at the trigger and seed its prompt from the Triage (on a schedule) preset. Choose the cheapest model on your provider that calls tools reliably, and set a budget.
4. Create the main agent. Seed its prompt from Root cause after a handoff and set its trigger to No trigger: only a handoff starts it.
5. Add the handoff. Open the triage agent, and under When it concludes choose Add a handoff. Pick investigate, the main agent and a cooldown of 30m.
6. Raise the daily run limit. At a 10 minute interval the triage agent runs 144 times a day, above the default of 50 runs per agent per day. Raise it under Settings → Agents or with TARES_AGENT_DAILY_CAP.

The same setup in the catalog:

triggers:
  - name: watch-api
    sources: [api_logs]
    key_field: service
    condition:
      every: 10m                          # at least 1m
      summary_by: [service, status_code]  # labels to count
 
agents:
  - name: triage
    trigger: watch-api
    concludes: true
    verdicts:
      - { verdict: investigate, when: one entity looks like a problem and needs a closer look }
    prompt: ...             # from the Triage (on a schedule) preset
    handoffs:
      - { verdict: investigate, agent: rca, cooldown: 30m }   # at most two root-cause runs per hour per entity
  - name: rca               # no trigger: it runs only when handed off
    concludes: true
    verdicts:
      - { verdict: rca, when: you found what is failing and why }
      - { verdict: resolved, when: it was noise, or it is already over }
    prompt: ...             # from the Root cause after a handoff preset

Use the handoff or a trigger over findings to wake the main agent, not both, or it runs twice for each flag.

What you get

• Coverage without a rule. Every interval, an agent looks at your API’s error codes, including failures nobody wrote an alert for.
• Quiet most of the time. Normal ticks end as “no finding” rows with a one-line reason and the cost of the run.
• One chain per incident. When triage flags a service, the root cause appears on the project Overview under What the agents found as one card with a headline and the next step. Opening it shows the tick, the triage finding and the root cause in order.
• Controlled spend. A small model reading a summary costs well under a cent a run. The main agent runs only when triage flags something, at most once per cooldown per entity.
• An easy off switch. Disable the triage agent or pause its schedule trigger. The main agent then goes quiet too, and alert-driven agents keep running as before.

When something else fits

• If you already have a precise rule for the failure, such as a specific alert on error rate, an alert-driven agent reacts the moment it fires. See AI SRE.
• If you need a condition before waking the main agent, such as several investigate findings in a window or findings from more than one agent, use a trigger over findings instead of a handoff. Watch, then escalate shows how.

Related