How do I define a trigger that fires when the 5xx rate crosses a threshold?

In Tares, you define a trigger whose condition aggregates a numeric field such as rate_5xx over a window and compares it with a predicate like ”> 1.0”. Set group_by to key_value so each service fires on its own. Tares then wakes every subscribed agent with the entity’s correlated timeline.

How it works

A trigger watches one or more sources and checks its condition as events arrive. When the condition holds, Tares POSTs one JSON body to each subscribed webhook. A Tares agent that a project wires to the trigger is woken as well. The full trigger model is on the Watch with triggers page.

The condition has four parts for a threshold:

• aggregate is one of count, sum, avg, max, min or any.
• field is the typed field you aggregate. At least one of the trigger’s sources must produce it, for example a Prometheus source that emits rate_5xx, labeled by service.
• predicate is the comparison. With max over rate_5xx and the predicate > 1.0, the trigger fires when the highest rate_5xx in the window exceeds 1.0.
• window is how far back the aggregate looks, such as 1m.

Two more keys decide who the condition applies to:

• key_field is the label that names the entity across your sources, such as service. Each source should carry it. If you leave it out, it defaults to the first source’s primary label.
• group_by: [key_value] evaluates the condition per entity, so each service fires independently.

filters are optional. Each one is a field, an op and a value, and only matching events count. The op is one of eq, neq, contains, gt, lt, gte or lte. The field is a label your sources carry, or one of event_type, source, text and key_value.

Set it up

1. Make sure a source produces a numeric 5xx field. A Prometheus source emitting rate_5xx, labeled by service, works. Give any log or metric source you add to the same trigger the same service label.
2. Add the trigger on the project’s page (the + next to Triggers), under Triggers → Add trigger, or in the catalog YAML. This example fires when the maximum rate_5xx for a service in production exceeds 1.0 within one minute:

triggers:
  - name: error_spike
    project: checkout watch   # optional; the Default project when left out
    sources: [checkout_metrics, checkout_logs]
    key_field: service        # optional; default: the first source's primary label
    filters:                  # optional; only these events count
      - { field: env, op: eq, value: prod }
    condition:
      aggregate: max          # count | sum | avg | max | min | any
      field: rate_5xx
      predicate: "> 1.0"      # fire when max(rate_5xx) over the window exceeds 1.0
      window: 1m
      group_by: [key_value]   # evaluated per entity (per service)
    emit:
      kind: error_spike
      context_window: 15m     # how much timeline to hand the agent
    cooldown: 5m              # minimum gap between firings for the same entity

3. Set emit.context_window to how much timeline the agent should receive. In the example it is 15 minutes.
4. Set cooldown to the minimum gap between firings for the same entity. A flapping 5xx rate then does not fire repeatedly.
5. Subscribe an agent. A Tares agent wakes once you turn it on in a project that wires it to the trigger. An external agent registers a webhook over MCP with the subscribe tool. For example: use tares to subscribe to the error_spike trigger and deliver to https://my-agent.example.com/hook. You can also register with POST /subscribe.
6. Over MCP, an agent can create and change the same trigger with create_trigger and update_trigger, which take the same fields.

What you get

• Per-service firing. With group_by: [key_value], one noisy service does not hide or trigger another.
• Context in the dispatch. The payload is the entity’s correlated timeline across the trigger’s sources over context_window: the deploy, the logs and the metrics around the spike. The agent needs no follow-up call.
• A dispatch body you can route. It carries dispatch_id, trigger, project, projects, kind, key, fired_at, payload and run_ids. Here key is the entity, such as api-server.
• A record of every firing. Firings are logged even with zero subscribers. You see them in the console under Firings, on the trigger’s page and on the project’s Activity.
• Safe delivery. Delivery is at-least-once, so dedupe on dispatch_id. Answer with a 2xx. 5xx responses and transport errors are retried with backoff, and a 4xx is recorded as a failed delivery and not retried.

To cover other incident shapes, change the aggregate and field. For example, max(p99_ms) > 1000 covers latency, and max(rate_5xx) > 1.0 covers an error storm.

When something else fits

If you want to review your sources on a clock instead of reacting to a threshold, give the trigger a schedule:

condition: {every: 10m}

The trigger then fires once per interval for all its events and hands over a summary of the window. The Watch, then escalate guide covers this pattern.

If your 5xx signal is not yet a number, first bring it in through a source that produces a typed field, such as the Prometheus connector, so the condition has something to aggregate. Use filters when only part of the traffic should count, for example env equal to prod. Use a longer window or a longer cooldown when short spikes should not wake an agent.

Related