How do I wake an AI SRE once when Alertmanager alert groups report slow database queries?

Tares wakes your AI SRE once per entity when you add a trigger with a cooldown on the Alertmanager source that carries your slow query alerts. Alertmanager keeps repeating the alert, the cooldown holds back repeat firings for the same entity, and each firing hands the agent one correlated timeline. With send_resolved on, resolved alerts are recorded as events too.

How it works

• Alertmanager POSTs a batch to the source’s ingest URL. Tares fans the alerts array out into one event per alert. event_type is the alert name, text is the summary, and the alert labels become Tares labels plus a synthetic status of firing or resolved.
• The entity key comes from a label, in this order: namespace, service, job, instance, or a declared primary label. You can override it with key_label on the source.
• A trigger evaluates its condition as events arrive. With group_by: [key_value] it evaluates per entity, so each namespace or service fires on its own.
• cooldown is the minimum gap between firings for the same entity. While Alertmanager re-sends the same slow query alert, the cooldown keeps it from starting a new investigation each time.
• Every alert, firing or resolved, is stored as an event on the source whether or not the trigger fires. When the trigger fires, Tares hands the agent the entity’s timeline across the trigger’s sources over context_window.

This is the pattern behind the AI SRE guide: Alertmanager decides what is alerting, and Tares decides when an agent needs to look.

Set it up

1. Create the source. Go to Sources → Add source → Alertmanager alerts, name it and click Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Either way you get an ingest URL.

2. Add the receiver to your alertmanager.yml and reload Alertmanager. Keep send_resolved: true so resolved alerts reach Tares.

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If your Tares instance has auth on, the Alertmanager receiver sends a Bearer Authorization header holding an ingest-scoped API key. The source page shows the auth block already filled in. Without auth, skip this step. See the Alertmanager connector page for the details.

4. Add the trigger on the project’s page, under Triggers → Add trigger, or in the catalog YAML. This one counts firing slow query alerts per entity and holds repeats back for 30 minutes. Replace the value with the alert name from your Alertmanager rule, and db-alerts with your source name.

triggers:
  - name: slow_queries
    sources: [db-alerts]
    filters:
      - { field: event_type, op: eq, value: PostgresSlowQueries }
      - { field: status, op: eq, value: firing }
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: slow_queries
      context_window: 15m
    cooldown: 30m

5. Wire the agent. A Tares agent is woken by the trigger in a project that wires it there, once you turn it on. An external agent registers a webhook with the subscribe tool, and you can ask it in plain words to subscribe to the slow_queries trigger.

6. Verify with a test batch, then check Recent events on the source.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

The source’s Fields view shows the alert labels, so you can promote namespace or service to correlate on.

What you get

• One investigation per entity per cooldown window, however often Alertmanager repeats the alert.
• Independent firing per entity, so a slow query alert in one namespace does not hold back another.
• Every alert stored as an event with its full payload, including the fingerprint, and a status label that tells firing from resolved.
• A dispatch whose payload is the entity’s correlated timeline over context_window, so the agent starts with context and makes no follow-up call.
• A log of every firing and each subscriber’s delivery status under Firings, on the trigger’s page and in the project’s Activity. Delivery is at-least-once, so dedupe on dispatch_id in your own endpoint.

The status filter in the example keeps a resolved alert from waking the agent. Resolved events are still recorded on the source. Filters also decide which events are handed over, so the dispatched timeline then holds the firing events. If you want resolved events in the handed timeline, drop the status filter. A resolved event can then satisfy the condition, and the cooldown still limits how often it fires.

When something else fits

• To look at the database on a clock instead of on an alert, give the trigger a schedule with condition: {every: 10m}, as in Watch, then escalate.
• If you want Prometheus rules polled directly without Alertmanager’s grouping and silencing, use the prometheus_alerts connector instead. Choose the Alertmanager source when you want its grouping, silencing, resolved semantics and real-time delivery.

Related