How do I stop a flood of alerts from Prometheus from waking an AI SRE again and again?

Tares stops an alert flood from waking your AI SRE repeatedly with three settings: severity and environment filters, a per-entity cooldown, and a trigger keyed by service. The Prometheus alerts source already emits each alert once, and the trigger decides when an agent wakes. Each service fires independently, and a flapping service waits out its cooldown instead of paging again.

How it works

Two layers work together, one on the source and one on the trigger.

• The source emits each alert once. The Prometheus alerts connector polls Prometheus’s own alerts endpoint. It tracks state across polls, so an alert fires once and re-emits only when its state changes. When the alert clears, you get one resolved event, not a copy on every tick.
• The source can filter before ingest. The severities field takes a comma-separated list such as critical,warning. Blank means all. Pending alerts, the ones still waiting out their for: duration, are left out unless you turn on include_pending.
• The trigger decides when an agent wakes. A trigger watches the source and evaluates a condition as events arrive. Its filters narrow which events count, key_field names the entity, and cooldown sets the minimum gap between firings for the same entity.
• Grouping makes each service independent. With group_by: [key_value], the condition is evaluated per entity, so each service fires on its own. One noisy service does not use up the cooldown of another.

Set it up

1. Add the Prometheus alerts source. Under Sources → Add source → Prometheus alerts, enter the url and any auth, then Discover. Set severity to the levels you want to be woken for.

   - name: alerts
     connector: prometheus_alerts
     poll: 30s
     config:
       url: http://localhost:9090
       severities: critical,warning   # blank = all
       labels:
         - { name: severity, field: labels.severity, primary: true }
         - { name: state, field: state }

2. Promote the labels you want to key and filter on. Discover only proposes severity and state, because an alert’s own labels exist only while it fires. Once real alerts have been ingested, open the Fields tab and add labels such as service and env. Each becomes an entry under labels, in the same form as the two above.

3. Create the trigger on the project’s page (+ next to Triggers), under Triggers → Add trigger, or in the catalog YAML. This one counts alerts per service in prod and waits five minutes before the same service can fire again:

   triggers:
     - name: critical_alerts
       sources: [alerts]
       key_field: service
       filters:
         - { field: env, op: eq, value: prod }
       condition:
         aggregate: count
         predicate: "> 0"
         window: 5m
         group_by: [key_value]
       emit:
         kind: critical_alerts
         context_window: 15m
       cooldown: 5m

4. Tune the three controls to your noise level:

   • Filters use {field, op, value} with op one of eq, neq, contains, gt, lt, gte or lte. The field can be a label your source carries or one of event_type, source, text and key_value.
   • key_field is the label that names the entity. If you leave it out, it defaults to the first source’s primary label.
   • cooldown is the minimum gap between firings for the same entity. Lengthen it for services that flap.

5. Subscribe the agent. A Tares agent is woken by the trigger in a project that wires it there, once you turn it on. An external agent registers a webhook with the subscribe tool.

6. Verify. Open the source: Recent events shows one event per active alert, plus resolved events as alerts clear. Fields shows the labels your alerts carry.

What you get

• One wake-up per service, not per alert. The condition is evaluated per entity, and the cooldown stops a flapping condition from firing repeatedly for the same service.
• Only the alerts you care about. Severity filters on the source and environment filters on the trigger keep low-priority and non-prod alerts from reaching the agent.
• No repeated copies of the same alert. Each alert fires once and re-emits only on a state change, with a single resolved event when it clears.
• Context in the dispatch. When a trigger fires, the agent receives the entity’s correlated timeline over context_window, so it starts with the surrounding events instead of making a follow-up call.
• A record of every firing. Every firing is logged, even with zero subscribers. You can see firings and per-subscriber delivery status in the console under Firings, on the trigger’s page, and on the project’s Activity.
• Safe retries. Delivery is at-least-once, so dedupe on dispatch_id if your endpoint can see the same dispatch twice.

When something else fits

• You want a look at the alerts on a clock, not a wake-up per condition. Give the trigger a schedule with condition: {every: 10m}. It fires once per interval for all its events and hands over a summary of the window. See Watch with triggers and Watch, then escalate.
• You need to see pending alerts early. Turn on include_pending on the source. Expect more events, so keep the trigger’s filters and cooldown tight.
• Your alerts reach you through Alertmanager. Use the alertmanager connector, which receives alerts at a Tares ingest URL, instead of polling Prometheus. The trigger settings above work the same way on its events.
• You want the agent to see everything. Leave severities blank and drop the filters. Keep the cooldown, so each service still fires only once per gap.

Related