What cooldown should I set so an AI SRE does not run twice for one incident?

In Tares, you set cooldown on a trigger as the minimum gap between firings for the same entity, for example cooldown: 5m. A flapping condition then cannot fire repeatedly for that entity. Pick the gap you accept between two wake-ups of your agent for one service, then check the Firings list and adjust.

How it works

A trigger watches one or more sources and checks its condition as events arrive. When the condition holds, Tares sends one JSON dispatch to each subscriber and wakes the agents wired to the trigger.

The cooldown is one field on that trigger:

• Minimum gap. cooldown is the minimum gap between firings for the same entity. Two firings for one entity are never closer together than the cooldown.
• Per entity. With group_by: [key_value] the condition is evaluated per entity, so each service fires independently. The cooldown applies to the same entity, so one service firing does not set the gap for another.
• Flapping. The cooldown exists to stop a flapping condition from firing repeatedly for the same entity.
• Separate from the window. The condition window sets how much data the aggregate covers. The context_window sets how much timeline the agent receives. The cooldown is a third value, and you size it on its own.

Set it up

1. Open the trigger on the project’s page, under Triggers → Add trigger, or edit the catalog YAML.
2. Set key_field to the label that names the entity, such as service, and set group_by: [key_value] so each entity is evaluated and rate-limited on its own.
3. Set cooldown to the gap you want between two wake-ups for one entity. The example below uses 5 minutes with a 1 minute window.

triggers:
  - name: error_spike
    project: checkout watch   # optional; the Default project when left out
    sources: [checkout_metrics, checkout_logs]
    key_field: service        # optional; default: the first source's primary label
    filters:                  # optional; only these events count
      - { field: env, op: eq, value: prod }
    condition:
      aggregate: max          # count | sum | avg | max | min | any
      field: rate_5xx
      predicate: "> 1.0"      # fire when max(rate_5xx) over the window exceeds 1.0
      window: 1m
      group_by: [key_value]   # evaluated per entity (per service)
    emit:
      kind: error_spike
      context_window: 15m     # how much timeline to hand the agent
    cooldown: 5m              # minimum gap between firings for the same entity

4. Let the trigger run, then open the Firings list on the trigger’s own page. Every firing is logged there, so you can see how often one entity fired and how far apart the firings were.
5. Change cooldown if two firings that you count as one incident sit closer together than you want, or if the gap is longer than you need.
6. Make your endpoint safe against redelivery. Delivery is at-least-once, so one firing can reach your agent more than once. Dedupe on dispatch_id, which is unique to each firing.

An agent can make the same change over MCP with update_trigger, which takes the same fields.

What you get

• A clear rule for repeats. Firings for one entity are never closer than the cooldown, and a flapping condition does not keep waking your agent.
• Independent entities. Each service has its own gap, so a noisy service does not hold back an alert for another.
• A record to tune against. Every firing is logged, even with zero subscribers, and each dispatch has its own page with the delivery outcomes and the exact payload sent. You can size the cooldown from what actually fired.
• Context in every dispatch. The payload carries the entity’s correlated timeline over context_window, so the agent that runs has the deploy, logs and metrics without a follow-up call.
• No double runs from retries. Deduping on dispatch_id covers the case where Tares delivers the same firing twice. This is separate from the cooldown, and you need both.

When something else fits

A cooldown is not the only way to keep an agent from running more than you want.

• Fix the condition first. Choose the aggregate and field to match the incident shape, for example max(p99_ms) > 1000 for latency and max(rate_5xx) > 1.0 for an error storm. A condition that fits the incident gives the cooldown less to damp.
• Use a schedule for a periodic look. If you want the sources checked on a clock and not on a condition, give the trigger a schedule with every: 10m. It fires once per interval for all its events and hands over a summary of the window.
• Dedupe for repeated delivery. If the same dispatch_id arrives twice, that is a redelivery of one firing, not a second firing. The cooldown does not apply to it, so handle it in your endpoint.

The full set of trigger fields, subscriptions and delivery rules is in Watch with triggers.

Related