How do I route Alertmanager alert groups about crash loops to an AI SRE?

Tares lets you send Alertmanager alerts to an alertmanager source and add a trigger that matches only your crash loop alert, so only that symptom wakes the agent. Alertmanager keeps grouping and routing. Tares records each alert as its own event and the trigger filters on the alert name and the firing status. You name the trigger after the symptom, so each firing and the agent it wakes read as crash loops.

How it works

• Alertmanager evaluates, groups and dedupes alerts, then POSTs a batch to the ingest URL of your Tares alertmanager source. Tares splits the batch into one event per alert.
• Each event has event_type set to the alert name, text set to the alert summary, the alert labels as Tares labels, and a synthetic status of firing or resolved. The full alert, including its fingerprint, stays in the payload.
• The entity key comes from a label, checked in the order namespace, service, job, instance, unless you declare a primary label or set key_label. Events for the same workload therefore land on the same timeline.
• A trigger watches the source. Its filters keep only events whose alert name is your crash loop alert and whose status is firing. Its condition decides when that match is enough to fire, and the cooldown stops repeat wake-ups.
• When the trigger fires, the agent subscribed to it wakes and reads the correlated timeline. The AI SRE guide walks through this chain end to end.

Set it up

1. Create the source. Ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Or open Sources → Add source → Alertmanager alerts, name it and click Create. The source page shows a receiver block with your ingest URL filled in.
2. Add the receiver to your alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

Keep send_resolved: true so resolved alerts arrive too. The trigger below ignores them, and they still show on the timeline. 3. If your Tares has auth on (tares up --auth), the sender uses a Bearer Authorization header holding an ingest-scoped API key. On any other instance you need no key. The source page shows the auth block when your instance requires a token. 4. Check that alerts arrive. Post a test batch, then open Recent events and Fields on the source page:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

Fields lists the alert labels, so you can confirm the alert name and promote namespace or service as the label to correlate on. 5. Add a trigger that matches only the crash loop symptom. Replace the value with the alertname your own crash loop rule uses, and set sources to the name you gave the source:

- name: crash-loops
  sources: [alerts]
  filters:
    - { field: event_type, op: eq, value: KubePodCrashLooping }
    - { field: status, op: eq, value: firing }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 5m

6. Subscribe an agent to the crash-loops trigger. It can be a Tares agent, your own agent service over a webhook, or a Slack channel. The AI SRE guide shows each option. Write the agent prompt around the symptom: take the first look at the crash looping workload, tie it to specific evidence lines and suggest a next action.
7. Test the filter both ways. The PostgresDown batch above must not fire crash-loops. Post the same batch with your crash loop alert name in alertname and it should fire. The firing then appears on the trigger’s page and under Firings.

What you get

• Only crash loop alerts wake the agent. Other alerts from the same Alertmanager still become events, so they stay on the timeline as evidence without starting a run.
• The trigger is named crash-loops, so every firing and the run it wakes are identified by the symptom, not by a generic alert stream.
• The agent reads one correlated timeline for the affected entity. Alertmanager alerts sit next to your other sources when they share the same label, such as service.
• Alertmanager keeps owning grouping, silencing and resolved tracking. Tares records what it pushes and does not re-threshold anything.
• The cooldown keeps one noisy loop from waking the agent again and again.

When something else fits

• If your alerts do not come from Alertmanager, the webhook connector accepts arbitrary JSON, and you can filter and trigger on its fields the same way.
• If you want every alert to wake an agent regardless of symptom, drop the alert name filter and keep only the status filter. Details of the receiver itself are on the Alertmanager connector page.

Related