How do I wake an AI SRE once when Alertmanager alert groups report error spikes?

Tares wakes your AI SRE once per entity when you pair the alertmanager connector with a trigger that sets a cooldown. Alertmanager alerts arrive as one event each, the trigger fires on the first, and the cooldown holds back repeats for the same entity. Resolved alerts are still recorded as events on that entity’s timeline.

How it works

• One event per alert. Alertmanager POSTs a batch to the source’s ingest URL. Tares splits the alerts array into one event each. The event_type is the alert name, text is the summary, and the alert labels become Tares labels plus a synthetic status of firing or resolved.
• One entity per alert group member. The entity key comes from a label, checked in the order namespace, service, job, instance. You can set key_label on the source to pick another label.
• A trigger watches the source. It evaluates its condition as events arrive. With group_by: [key_value] it evaluates per entity, so each service or namespace fires on its own.
• The cooldown is the minimum gap between firings for the same entity. A repeating alert for prod fires one investigation, and the next firing for prod has to wait until the cooldown has passed. Other entities are not held back.
• Resolved events are recorded. With send_resolved: true, Alertmanager sends the resolved alert and Tares stores it with status set to resolved, next to the firing events and your other sources.

The Alertmanager connector page covers the receiver. The triggers guide covers conditions, cooldowns and delivery.

Set it up

1. Create the source. In the console go to Sources → Add source → Alertmanager alerts, name it and select Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” You get an ingest URL.
2. Add the receiver to alertmanager.yml and reload Alertmanager. Keep send_resolved: true so resolved alerts reach the timeline.

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If your instance runs with auth on (tares up --auth), the sender uses a Bearer Authorization header holding an ingest-scoped API key. The source page shows the auth block already filled in. Without auth on, you need no key.
4. Send a test alert, then check Recent events and Fields:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

5. Add the trigger on the project’s page, under Triggers → Add trigger, or in the catalog YAML. Replace alertmanager_alerts with your source name, and the event_type value with your error-spike alert name:

triggers:
  - name: error_spike
    sources: [alertmanager_alerts]
    key_field: namespace
    filters:
      - { field: event_type, op: eq, value: HighErrorRate }
      - { field: status, op: eq, value: firing }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 1m
      group_by: [key_value]
    emit:
      kind: error_spike
      context_window: 15m
    cooldown: 5m

6. Wake the agent. A Tares agent in the project wakes on this trigger once you turn it on there. An external agent registers with the subscribe tool: “Use tares to subscribe to the error_spike trigger and deliver to https://my-agent.example.com/hook.”

Set cooldown longer than the time your alert normally keeps repeating, so one incident stays one investigation.

What you get

• One dispatch per entity per cooldown window. Each dispatch carries the dispatch_id, the trigger, the entity key and the entity’s correlated timeline over context_window.
• The full Alertmanager alert, including its fingerprint, kept in the event payload.
• Resolved alerts stored as events with status set to resolved, so you can see when each incident ended.
• Every firing logged under Firings, on the trigger’s page and on the project’s Activity, even with zero subscribers.
• Delivery you can rely on. It is at-least-once, so dedupe on dispatch_id. A 2xx acknowledges, 5xx responses are retried with backoff, and a 4xx is recorded as failed and not retried.

The status filter in step 5 narrows which events count and which are handed over. With it, resolved events stay on the source’s timeline, but the dispatch payload holds firing events only. Leave that filter out if you want resolved events inside the payload. A resolved event then counts toward the condition too, and it can fire once the cooldown has passed.

When something else fits

• If you want a regular summary instead of a reaction to alerts, give the trigger a schedule with condition: {every: 10m}. See Watch, then escalate.
• If you want Prometheus rules polled directly rather than Alertmanager’s grouping, silencing and resolved handling, use the prometheus_alerts connector.
• If you have a numeric field such as rate_5xx on a metrics source, a threshold like max(rate_5xx) > 1.0 fits better than counting alerts.

Related