What trigger condition catches error spikes early?

In Tares, use a trigger that takes the max of a typed error-rate field such as rate_5xx over a short window, evaluated per entity. Set a predicate such as ”> 1.0”, group by key_value, and add a cooldown. The condition is checked as events arrive, so each service fires on its own as soon as its rate crosses the line.

How it works

A trigger watches one or more sources and evaluates its condition as events are ingested. For error spikes, the condition has five parts:

• aggregate: max takes the highest value of the field inside the window. The other aggregates are count, sum, avg, min and any.
• field: rate_5xx is the typed numeric field for this symptom. A Prometheus source can emit it, labeled by service.
• predicate: "> 1.0" is the threshold. The trigger fires when max(rate_5xx) over the window exceeds 1.0.
• window: 1m is the span the aggregate covers. A short window means a spike shows up within a minute of events arriving.
• group_by: [key_value] evaluates the condition per entity, so each service fires independently and one noisy service does not hide another.

cooldown sets the minimum gap between firings for the same entity. It stops a flapping condition from firing again and again for one service.

The key_field names the entity across your sources, such as service. If you leave it out, it defaults to the first source’s primary label. Each source should carry that label. Optional filters narrow which events count. Each filter is a field, an op (eq, neq, contains, gt, lt, gte or lte) and a value.

Set it up

1. Make sure one of the trigger’s sources produces a numeric field. For example, a Prometheus source that emits rate_5xx and carries a service label.
2. Add the trigger on the project’s page (+ next to Triggers), under Triggers → Add trigger, or in the catalog YAML:

triggers:
  - name: error_spike
    project: checkout watch   # optional; the Default project when left out
    sources: [checkout_metrics, checkout_logs]
    key_field: service        # optional; default: the first source's primary label
    filters:                  # optional; only these events count
      - { field: env, op: eq, value: prod }
    condition:
      aggregate: max          # count | sum | avg | max | min | any
      field: rate_5xx
      predicate: "> 1.0"      # fire when max(rate_5xx) over the window exceeds 1.0
      window: 1m
      group_by: [key_value]   # evaluated per entity (per service)
    emit:
      kind: error_spike
      context_window: 15m     # how much timeline to hand the agent
    cooldown: 5m              # minimum gap between firings for the same entity

3. Set context_window to how much timeline you want handed over. In the example it is 15 minutes, so the agent sees the deploy, logs and metrics around the spike.
4. Subscribe an agent. A Tares agent is woken by the trigger once you turn it on in a project that wires it up. An external agent registers a webhook over MCP with the subscribe tool:

Use tares to subscribe to the error_spike trigger and deliver to https://my-agent.example.com/hook.

5. Make your endpoint accept a POST and reply with a 2xx. Dedupe on dispatch_id, because delivery is at-least-once. Tares retries 5xx responses and transport errors with backoff, and records a 4xx as a failed delivery without retrying.

Over MCP, an agent can create the same trigger with create_trigger and update_trigger, which take the same fields.

What you get

• Early detection per service. A short window and a max aggregate fire on the worst reading, and the condition is checked as events are ingested.
• Independent firings. Grouping by key_value means each service has its own state and its own cooldown.
• Context in the dispatch. Each firing POSTs one JSON body with dispatch_id, trigger, kind, key, fired_at and a payload. The payload is the entity’s correlated timeline across the trigger’s sources over context_window, so the agent needs no follow-up call.
• A full record. Every firing is logged, even with zero subscribers. You see firings and per-subscriber delivery status in the console under Firings, on the trigger’s page and in the project’s Activity. Each dispatch has its own page with the delivery outcome and the exact payload sent.
• Fewer repeat alerts. The 5 minute cooldown in the example keeps one ongoing spike from waking your agent on every event.

Tune the threshold and window to your traffic. Pick the aggregate and field for the incident shape you care about: max(rate_5xx) > 1.0 for an error storm, or max(p99_ms) > 1000 for latency.

When something else fits

• You want a check on a clock, not a condition. Give the trigger a schedule with condition: {every: 10m}. It fires once per interval for all its events and hands over a summary of the window. See Watch with triggers for the schedule form.
• You only need to count events. Use aggregate: count with a predicate such as ”> 0” and a window, and narrow the events with filters. This suits discrete events like a merged pull request more than a continuous rate.
• You want a trend rather than a peak. Use avg instead of max. It smooths out one-off readings, at the cost of firing later on a sharp spike.
• One report for the whole project. An outside agent can join the project and be woken by all of its triggers at once, instead of subscribing to each trigger.

Related