What trigger condition catches 5xx errors on an API early?

Tares catches 5xx errors early with a trigger that takes the max of a numeric field such as rate_5xx over a short window, per service. A condition like max(rate_5xx) > 1.0 over 1m, grouped by key_value with a cooldown, fires as soon as the error rate spikes for one API. Each service fires on its own, and the agent you subscribe is woken with that service’s correlated timeline.

How it works

A trigger watches one or more sources and checks its condition as each event arrives. For a 5xx symptom, the condition has five parts:

• aggregate: max takes the highest value of the field inside the window. The other aggregates are count, sum, avg, min and any. Max reacts to a single bad reading, which suits an early warning.
• field: rate_5xx is the typed numeric field to aggregate. A Prometheus source can emit it, labeled by service. At least one of the trigger’s sources must produce a numeric field, because triggers aggregate a typed field.
• predicate: "> 1.0" is the test applied to the aggregate. The trigger fires when max(rate_5xx) over the window exceeds 1.0.
• window: 1m is how much recent data the aggregate covers. A short window means the trigger sees a spike within a minute.
• group_by: [key_value] evaluates the condition per entity, so each service fires independently. key_field names that entity across your sources, for example service, and defaults to the first source’s primary label.

The cooldown sets the minimum gap between firings for the same entity. It stops a flapping error rate from firing again and again for one API.

When the condition holds, Tares POSTs a JSON dispatch to each subscriber. The payload carries the entity’s correlated timeline across the trigger’s sources over context_window: the deploy, the logs and the metrics around the spike. The agent gets context without a follow-up call.

Set it up

1. Make sure one of your sources emits a numeric 5xx field. For example, a Prometheus source emitting rate_5xx, labeled by service, with each source carrying the label you group on.
2. Add the trigger on the project’s page (+ next to Triggers), under Triggers → Add trigger, or in the catalog YAML. This is the error storm shape:

triggers:
  - name: error_spike
    project: checkout watch   # optional; the Default project when left out
    sources: [checkout_metrics, checkout_logs]
    key_field: service        # optional; default: the first source's primary label
    filters:                  # optional; only these events count
      - { field: env, op: eq, value: prod }
    condition:
      aggregate: max          # count | sum | avg | max | min | any
      field: rate_5xx
      predicate: "> 1.0"      # fire when max(rate_5xx) over the window exceeds 1.0
      window: 1m
      group_by: [key_value]   # evaluated per entity (per service)
    emit:
      kind: error_spike
      context_window: 15m     # how much timeline to hand the agent
    cooldown: 5m              # minimum gap between firings for the same entity

3. Narrow the events with filters. Each filter is a field, an op and a value, where op is one of eq, neq, contains, gt, lt, gte or lte. The example keeps only production events.
4. Tune the numbers to your API. Lower the predicate or shorten the window to catch a spike sooner, and raise the cooldown if a noisy service pages too often.
5. Subscribe an agent. A Tares agent is woken by the trigger in a project that wires it to that trigger. An external agent registers a webhook over MCP with the subscribe tool, for example: Use tares to subscribe to the error_spike trigger and deliver to your agent’s URL. subscribe(trigger, url) returns a subscription id.
6. Acknowledge each dispatch with a 2xx. Deliveries are at-least-once, so dedupe on dispatch_id. Responses in the 5xx range and transport errors are retried with backoff, and a 4xx is recorded as a failed delivery and not retried.

An agent can create the same trigger over MCP with create_trigger and update_trigger, which take the same fields.

What you get

• An alert on the first minute of an error storm, not after a long average has caught up.
• One firing per affected service, because the condition is evaluated per entity. A 5xx spike on one API does not fire for the others.
• A dispatch with the entity’s correlated timeline, so the agent sees the deploy, the logs and the metrics around the spike.
• A cooldown that keeps a flapping error rate from firing repeatedly for the same service.
• A record of every firing, even with zero subscribers, in the console under Firings, on the trigger’s own page and on the project’s Activity. Each dispatch has its own page with the delivery outcomes and the exact payload that was sent.

For how triggers fit into incident response as a whole, see AI SRE.

When something else fits

• Pick different aggregates and fields for different incident shapes. Use max(p99_ms) > 1000 for latency, and keep max(rate_5xx) > 1.0 for an error storm.
• To review sources on a clock instead of on a condition, give the trigger a schedule with condition: {every: 10m}. It fires once per interval and hands over a summary of the window.

Related