What trigger condition catches high latency early?

Tares catches high latency early with a trigger on max(p99_ms) over a short window, evaluated per service, with a predicate such as > 1000 and a cooldown. The condition is checked as events are ingested, so the trigger fires as soon as the window’s max crosses your threshold. The agent you subscribe is woken with the service’s correlated timeline. This is one of the aggregate-and-predicate conditions described in Watch with triggers.

How it works

A trigger watches one or more sources and evaluates its condition as each event arrives. For latency, the condition has five parts:

• aggregate: how to combine values in the window. The options are count, sum, avg, max, min and any. For latency, max reacts to a single slow stretch, where avg smooths it out.
• field: the typed numeric field to aggregate. For latency this is a field such as p99_ms. At least one of the trigger’s sources must produce it.
• predicate: the test on the aggregate, for example > 1000. The trigger fires when max(p99_ms) over the window exceeds 1000.
• window: how much recent time the aggregate covers, for example 1m.
• group_by: [key_value] evaluates the condition per entity, so each service fires on its own.

The entity comes from key_field, the label that names it across your sources, such as service. If you leave it out, it defaults to the first source’s primary label. Each source should carry that label.

A cooldown sets the minimum gap between firings for the same entity. A latency metric that hovers around the threshold does not page your agent over and over.

The guide’s own example of matching aggregate and field to the incident is max(p99_ms) > 1000 for latency, against max(rate_5xx) > 1.0 for an error storm.

Set it up

1. Check that a source carries a latency number. Your source must emit a numeric field such as p99_ms, labeled by the entity you care about, for example service.
2. Add the trigger. Use the + next to Triggers on the project’s page, or Triggers → Add trigger, or put it in the catalog YAML. Here checkout_metrics and checkout_logs stand in for your own source names:

triggers:
  - name: high_latency
    project: checkout watch   # optional; the Default project when left out
    sources: [checkout_metrics, checkout_logs]
    key_field: service        # optional; default: the first source's primary label
    filters:                  # optional; only these events count
      - { field: env, op: eq, value: prod }
    condition:
      aggregate: max          # count | sum | avg | max | min | any
      field: p99_ms
      predicate: "> 1000"     # fire when max(p99_ms) over the window exceeds 1000
      window: 1m
      group_by: [key_value]   # evaluated per entity (per service)
    emit:
      kind: high_latency
      context_window: 15m     # how much timeline to hand the agent
    cooldown: 5m              # minimum gap between firings for the same entity

3. Tune the threshold and window. A shorter window and a lower predicate fire sooner. A longer window and a higher predicate fire less often. Set cooldown to the gap you want between repeat firings for one service.
4. Subscribe an agent. A Tares agent is woken by the trigger in a project that wires it to that trigger, once you turn it on there. An external agent calls the subscribe tool over MCP with the trigger name and your endpoint URL. It returns a subscription id. You can also register with POST /subscribe.
5. Receive the dispatch. When the condition holds, Tares POSTs one JSON body to each subscriber. Acknowledge with a 2xx. Delivery is at-least-once, so dedupe on dispatch_id.

An agent can create the same trigger over MCP with create_trigger and update_trigger, which take the same fields.

What you get

• A per-service signal. Because of group_by: [key_value], a slow checkout service fires without touching a healthy one.
• Context with the wake-up. The dispatch payload is the entity’s correlated timeline across the trigger’s sources over context_window: the deploy, the logs and the metrics around the spike. Your agent starts with context and does not need a follow-up call.
• Controlled noise. The cooldown keeps a flapping latency metric from firing repeatedly for the same entity.
• A record of every firing. Every firing is logged, even with zero subscribers. You see firings and per-subscriber delivery status in the console under Firings, on the trigger’s own page, and on the project’s Activity. Each dispatch has a page with the delivery outcomes and the exact payload sent.

For how triggers fit into the wider approach to incident response, see AI SRE.

When something else fits

• If you want to look at the sources on a clock rather than on a threshold, give the trigger a schedule with condition: {every: 10m}. It fires once per interval and hands over a summary of the window.
• If the incident shape is an error storm and not slowness, aggregate a different field, such as max(rate_5xx) > 1.0.
• If no source produces a numeric latency field yet, add one first, because triggers aggregate a typed field.

Related