How do I route Alertmanager alert groups about slow database queries to an AI SRE?

Tares turns every alert Alertmanager sends into its own event, and a trigger with filters keeps only the slow database query alerts and wakes your agent. You point an Alertmanager webhook receiver at a Tares alertmanager source, then write a trigger that matches the alert names for that symptom. Other alerts still land on the timeline, but they do not wake the agent.

How it works

Alertmanager has already evaluated, grouped and deduped its alerts. When it POSTs a batch to the source’s ingest URL, Tares fans the alerts array out into one event per alert:

• event_type is the alert name.
• text is the alert’s summary.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including fingerprint, stays in the payload.

Because the alert name is the event type, a trigger can pick out one symptom by filtering on event_type. A second filter on status keeps resolved notices from waking the agent. The trigger’s condition counts the matching events in a window, and a cooldown stops repeated firings for the same symptom.

The agent subscribed to the trigger reads one correlated timeline for the service, so the slow-query alerts sit next to your metrics and logs when it investigates. Name the trigger after the symptom. Its firings are listed on the trigger’s page and under Firings, so each wake-up is tied to the slow query symptom. The AI SRE guide shows the whole chain of source, trigger and agent.

Set it up

1. Start Tares and open the console.

   tares up

2. Create the source. Go to Sources → Add source → Alertmanager alerts, name it and click Create. You can also ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The source page shows a copy-paste receiver block with the ingest URL filled in. The source looks like this:

   - name: db-alerts
     connector: alertmanager

3. Add the receiver to your alertmanager.yml and reload Alertmanager. Replace the placeholder with your ingest key.

   receivers:
     - name: tares
       webhook_configs:
         - url: http://localhost:8787/ingest/<ingest_key>
           send_resolved: true
   route:
     receiver: tares

   If your Tares has auth on (tares up --auth), Alertmanager must also send an Authorization header with the Bearer scheme holding an ingest-scoped API key. The source page includes that block when the instance requires a token.

4. Send a test batch and check Recent events. Fields shows the alert labels, so you can promote namespace or service to correlate on.

   curl -X POST http://localhost:8787/ingest/<ingest_key> \
     -H 'Content-Type: application/json' \
     -d '{"alerts":[{"status":"firing","labels":{"alertname":"SlowQueries","severity":"warning","namespace":"prod"},"annotations":{"summary":"Database queries are slow"},"startsAt":"2024-01-01T00:00:00Z"}]}'

5. Create a trigger that keeps only the slow query alerts. Use the alert names your rules actually send as the event_type values.

   - name: slow-db-queries
     sources: [db-alerts]
     filters:
       - { field: event_type, op: in, value: [SlowQueries, PostgresSlowQueries] }
       - { field: status, op: eq, value: firing }
     condition: { aggregate: count, predicate: "> 0", window: 5m }
     cooldown: 5m

6. Subscribe the agent that should wake. It can be a Tares agent, your own agent service over a webhook, or a Slack channel. Edit the agent’s prompt so it knows the trigger is about slow database queries.

The entity key comes from a label (namespace, then service, job, instance, or a primary label you declare). Keep the same key on your metric and log sources so the agent sees them on one timeline. To change it, set key_label on the source.

What you get

• Only alerts that match the slow query names, and are firing, wake the agent. Everything else is still recorded and correlated.
• Each wake-up shows under a trigger named for the symptom, so you can tell at a glance why the agent ran.
• Resolved notices stay on the timeline without waking the agent again.
• The cooldown stops a burst of repeated alerts from waking the agent over and over.
• The agent reads the alert events alongside metrics and logs for the same service, and its finding is written back to that timeline for the next investigation.
• Alertmanager keeps owning grouping, silencing and resolved tracking. Tares does not re-threshold anything.

When something else fits

• If you want Prometheus rules polled directly instead of routed through Alertmanager, use the prometheus_alerts connector. Alertmanager is the better choice when you want its grouping, silencing and real-time delivery.
• If the alerts come from a sender that is not Alertmanager, post them to a webhook source and filter the trigger the same way on the fields your payload carries.
• If the symptom shows up in a metric rather than in an alert, threshold the typed fields in a trigger instead of filtering on alert names.

Related