How do I wake an AI SRE once when alerts from Prometheus report error spikes?

Tares wakes your AI SRE once per entity when Prometheus alerts report an error spike. Ingest the alerts with the prometheus_alerts connector, then add a trigger with a cooldown and a key_field. The cooldown holds back repeat firings for the same service, and resolved events are still recorded when the alert clears.

How it works

• The prometheus_alerts connector polls Prometheus’s own alerts endpoint, every 30 seconds in the example below. Prometheus decides what counts as an alert. Tares needs no Alertmanager and no PromQL for this.
• Each alert becomes one event. Its event_type is the alert name, its text is the summary, and its labels become Tares labels plus a synthetic state of firing, pending or resolved. See Prometheus alerts for the full field list.
• State is tracked across polls. An alert fires once and re-emits only when its state changes, so a long-running alert does not produce a copy on every poll. When the alert disappears from Prometheus, Tares emits a resolved event.
• A trigger watches the source and checks its condition as events arrive. key_field names the entity, such as service, and group_by: [key_value] evaluates the condition per entity, so each service fires independently.
• cooldown sets the minimum gap between firings for the same entity. If an alert flaps, going from firing to resolved and back to firing, the cooldown stops it from waking the agent again inside that window.
• When the trigger fires, Tares pushes the entity’s correlated timeline over context_window to the subscribed agent. The agent gets the context without a follow-up call.

Set it up

1. Add the Prometheus alerts source. In the console, go to Sources → Add source → Prometheus alerts, enter the url and click Discover. Or use this catalog YAML:

   - name: alerts
     connector: prometheus_alerts
     poll: 30s
     config:
       url: http://localhost:9090
       severities: critical,warning   # blank = all
       labels:
         - { name: severity, field: labels.severity, primary: true }
         - { name: state, field: state }

2. Promote the label that names the entity. Dynamic labels such as namespace or service only exist while an alert is firing. Once real alerts have been ingested, open the source’s Fields tab and add the label you want to key on.

3. Define the trigger with a per-entity cooldown. Add it under Triggers → Add trigger or in the catalog YAML. This adapts the example from Watch with triggers:

   triggers:
     - name: error_spike
       sources: [alerts]
       key_field: service
       filters:
         - { field: state, op: eq, value: firing }
         - { field: severity, op: eq, value: critical }
       condition: { aggregate: count, predicate: "> 0", window: 5m, group_by: [key_value] }
       emit:
         kind: error_spike
         context_window: 15m
       cooldown: 30m

   Add a filter on event_type with the name of your error-rate alert if you want only that alert to count.

4. Wake the agent. A Tares agent is woken by the trigger in a project that wires it to that trigger, once you turn it on there. An external agent registers a webhook over MCP with the subscribe tool:

   Use tares to subscribe to the error_spike trigger and deliver to https://my-agent.example.com/hook.

5. Verify. Open the source and check Recent events for one event per active alert and for resolved events as they clear. Open the trigger’s page to see each firing and its per-subscriber delivery status.

What you get

• One investigation per service per cooldown window, not one per alert event.
• A dispatch with the entity’s correlated timeline over context_window: the alerts plus the logs, deploys and metrics from the trigger’s other sources, if you add them.
• A resolved event for every alert that clears, kept on the source so you can see when the spike ended. Filters narrow which events count and which are handed over. The state filter above keeps the trigger on firing alerts. Remove it if you want resolved events in the handed-over timeline. A resolved event then counts toward the condition too, and the cooldown still limits how often the same service wakes the agent.
• A logged record of every firing, even with zero subscribers, with its delivery outcome and the exact payload sent. Delivery is at-least-once, so dedupe on dispatch_id, and answer with a 2xx so Tares does not retry.

When something else fits

• If you want to look at the alerts on a clock instead of waking on each spike, give the trigger a schedule with condition: {every: 10m}. It fires once per interval and hands over a summary of the window. See Watch, then escalate.
• If you want the agent to react to a metric value such as max(rate_5xx) > 1.0 rather than to an alert Prometheus has already fired, use the Prometheus metrics connector and aggregate the numeric field in the trigger.
• If your alerts reach you through Alertmanager receivers, the alertmanager connector takes them at a Tares ingest URL instead of polling Prometheus.

Related