How do I route Alertmanager alert groups about a disk filling up to an AI SRE?

Tares turns every alert Alertmanager sends into its own event, and a trigger that filters on the alert name and on firing status wakes your AI SRE only for disk-filling alerts. You point an Alertmanager webhook receiver at a Tares alertmanager source, then write one trigger for the symptom. Other alerts still land on the timeline but do not wake the agent.

How it works

• Alertmanager POSTs a batch of alerts to the ingest URL of an alertmanager source. Tares fans the batch out into one event per alert.
• Each event has event_type set to the alert name and text set to the summary. The alert labels become Tares labels, plus a synthetic status of firing or resolved. The full alert, including its fingerprint, stays in the payload.
• A trigger watches the source, keeps only the events that match its filters, and fires when its condition holds. A filter is a field, an op and a value, and the condition is an aggregate, a predicate and a window.
• Filtering on event_type picks the symptom, because the event type is the alert name. Filtering on status keeps resolved notices from waking the agent.
• Each firing is listed under Firings and on the trigger’s own page, under the trigger’s name. Name the trigger after the symptom and the list reads as a record of disk alerts.

Tares records what Alertmanager pushes and correlates it with your other sources. Alertmanager keeps the grouping, deduplication and silencing. If an alert reaches you from a system other than Alertmanager, the webhook connector takes arbitrary JSON instead.

Set it up

1. Create the source. In the console go to Sources → Add source → Alertmanager alerts, name it and click Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The source page shows a receiver block with the ingest URL filled in. The source is the same as this entry in your catalog:

   - name: alerts
     connector: alertmanager

2. Add the receiver to your alertmanager.yml and reload Alertmanager:

   receivers:
     - name: tares
       webhook_configs:
         - url: http://localhost:8787/ingest/<ingest_key>
           send_resolved: true
   route:
     receiver: tares

   The sender uses a Bearer Authorization header holding an ingest-scoped API key, needed only when Tares has auth on (tares up --auth). The source page shows the auth block for you in that case.

3. Send a test alert and check Recent events and Fields on the source page. This confirms the alert name arrives as the event type and shows which labels you can correlate on:

   curl -X POST http://localhost:8787/ingest/<ingest_key> \
     -H 'Content-Type: application/json' \
     -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

4. Create the trigger for the symptom. Replace the alert names with the ones your Prometheus rules use for disk space. The filters keep only firing disk alerts, and the condition fires when at least one arrives within five minutes. The cooldown stops a flapping alert from waking the agent over and over:

   - name: disk-filling-up
     sources: [alerts]
     filters:
       - { field: event_type, op: in, value: [DiskFillingUp, DiskSpaceLow] }
       - { field: status, op: eq, value: firing }
     condition: { aggregate: count, predicate: "> 0", window: 5m }
     cooldown: 15m

5. Choose who gets woken. A Tares agent is defined by a prompt, so write one that says the job is a disk filling up: find which volume, how fast it grows, and what changed. You can also hand the firing to your own agent service or a Slack channel. The AI SRE guide walks through all three and shows a full trigger-to-agent chain.

6. Trigger a test alert. The firing appears under Firings with the name disk-filling-up, and the agent’s finding lands on the service’s timeline.

What you get

• Only disk alerts that are firing wake the agent. Other alerts from the same receiver are still recorded as events and stay on the timeline next to your metrics and logs for correlation.
• Resolved notices do not wake the agent, but they still arrive and show the alert clearing.
• A trigger named for the symptom, so Firings and the trigger page read as a history of disk alerts.
• One event per alert, with its labels and fingerprint kept in the payload, so the agent can see which host or volume fired.
• Alertmanager keeps owning grouping, silencing and resolved tracking. Tares does not re-threshold anything.

When something else fits

• If you want Tares to read the alerts straight from Prometheus without an Alertmanager receiver, the prometheus_alerts connector polls them instead. Use the alertmanager connector when you want Alertmanager’s grouping, silencing and real-time delivery.
• If the disk alerts come from a vendor other than Alertmanager, send them through the webhook connector and use the same kind of trigger.
• See the Alertmanager connector page for the receiver details.

Related