How do I wake an AI SRE once when alerts from Prometheus report crash loops?

Tares wakes an AI SRE from a trigger on your Prometheus alerts source, with a cooldown that sets the minimum gap between firings for the same entity. The source emits each alert once and adds a resolved event when it clears. Key the trigger on a label such as service and set the cooldown, so a repeating crash loop alert does not wake the agent on every poll.

How it works

The prometheus_alerts connector polls Prometheus’s own alerts endpoint, every 30 seconds by default. It keeps state across polls, so each alert fires once and re-emits only when its state changes. When the alert disappears, the source emits a resolved event. You get no repeated copy of the same alert on every tick.

Each alert becomes an event with these properties:

• event_type is the alert name.
• text is the alert summary.
• The alert’s labels become Tares labels, plus a synthetic state of firing, pending or resolved.

A trigger watches the source and evaluates a condition as events arrive. With group_by: [key_value] it evaluates per entity, so each service fires independently. The cooldown is the minimum gap between firings for the same entity, which stops a flapping condition from firing repeatedly. When the trigger fires, Tares pushes to the subscribed agent with the entity’s correlated timeline across the trigger’s sources over context_window. The agent starts with context and makes no follow-up call.

This is one part of the wider AI SRE setup.

Set it up

1. Add the alerts source. In the console, go to Sources → Add source → Prometheus alerts, enter the url and click Discover. Or use this catalog entry:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

2. Let a crash loop alert arrive, then open the source. The Fields tab shows the labels the alerts carry. Promote the one that names the entity, such as namespace or service. Discover only proposes severity and state, because an alert’s other labels exist only while it is firing.

3. Define the trigger. Set key_field to the label you promoted, so the entity is the service and not the default first primary label. Match the crash loop alert by name and keep only firing events:

triggers:
  - name: crash_loop
    sources: [alerts]
    key_field: service
    filters:
      - { field: event_type, op: contains, value: CrashLoop }   # use your alert's name
      - { field: state, op: eq, value: firing }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: crash_loop
      context_window: 15m
    cooldown: 30m

4. Choose the cooldown. It is the minimum gap between firings for the same entity, so a longer value means fewer wake-ups for a service that keeps crash looping. Pick one that fits how long your team takes to look at an incident.

5. Wake the agent. A Tares agent wakes when a project wires it to the trigger and you turn it on. An external agent registers a webhook with the subscribe tool over MCP. Make sure your endpoint acknowledges with a 2xx. Delivery is at-least-once, so dedupe on dispatch_id.

6. Verify. Open the source and check Recent events, then open the trigger’s page to see its firings and each subscriber’s delivery status.

What you get

• Each alert emits one event per state change, not one per poll tick.
• The trigger evaluates per entity, so one crash looping service does not hide another.
• A cooldown on the trigger sets the minimum gap between firings for the same entity.
• Each firing delivers the entity’s correlated timeline over your context_window.
• resolved events are emitted when Prometheus stops reporting the alert. Recent events on the source shows them as the alerts clear, so you can see when a crash loop ended.
• Every firing is logged, even with zero subscribers, under Firings, on the trigger’s page and in the project’s Activity.

When something else fits

• If you want to look at the alerts on a clock instead of on a condition, give the trigger a schedule such as condition: {every: 10m} and it fires once per interval with a summary of the window. See Watch with triggers.
• If the alerts reach you through Alertmanager and not by polling Prometheus, use the alertmanager connector. See Prometheus alerts for what the polling connector covers.

Related