What should Alertmanager alert groups about crash loops include for an AI SRE?

Tares works best when each crash loop alert carries a service label, a severity label, a summary annotation and a start time. Tares turns every alert in a group into its own event, with the alert name as the event type, the summary as the text and the labels as Tares labels. The service label lets the agent tie the alert to the metrics and logs for the same service.

How it works

Alertmanager has already evaluated, grouped and deduped the alerts. When it POSTs a batch to your Tares ingest URL, Tares fans the alerts array out into one event per alert:

• event_type is the alert name, so name the alert after the symptom, for example a name that says the service is crash looping.
• text is the alert’s summary annotation, so write a summary an agent can read on its own.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including fingerprint and startsAt, is kept in the payload.

So these four things are worth sending on every crash loop alert:

Tares derives the entity key from a label, checking namespace, then service, then job, then instance, or a label you declare as primary. If your alerts carry both namespace and service and you want the service to be the key, set the advanced key_label field to service. Keep the same label name on your other sources, because the AI SRE guide builds one timeline per service from sources that share a service label.

Set it up

1. Create the source. In the console go to Sources, then Add source, then Alertmanager alerts, name it and click Create. You can also ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” The source page shows a receiver block with the ingest URL filled in. The Alertmanager connector page has the full reference.
2. Add the receiver to your alertmanager.yml and reload Alertmanager. Set send_resolved: true so Tares also records when the crash loop clears.

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If your Tares instance runs with auth on (tares up --auth), the receiver sends a Bearer Authorization header holding an ingest-scoped API key in its http_config. Without auth, you do not need it.
4. In your alerting rules, put the service name and severity in the alert labels and the summary in the annotations. Add service to the labels even if namespace is already there, so the agent can match it to your other sources.
5. Send a test batch and check Recent events. The test below has a name, a severity, a namespace, a summary and a start time. Add a service label to it to match what your real alerts carry.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

6. Open Fields on the source. It lists the alert labels, so you can promote namespace or service to correlate on.

What you get

• One event per crash loop alert, with the alert name, summary text, labels and status, so the agent sees what fired and whether it resolved.
• The whole alert kept in the payload, so details such as fingerprint and startsAt stay available.
• A timeline per service where the alert sits beside your metrics and logs on one clock, so the agent can tie the crash loop to specific evidence lines.
• Alertmanager keeps owning grouping, silencing and resolved handling. Tares records what it pushes and correlates it with your other sources.
• A trigger on the alert source can wake an agent when an alert event arrives, as in the AI SRE guide.

When something else fits

• If you do not run Alertmanager and want Tares to read the alerts your Prometheus rules fire, use the prometheus_alerts connector, which polls every 30 seconds by default. Alertmanager’s grouping, silencing and real-time delivery come with the alertmanager connector.
• If the signal you need is the crash itself, such as the log lines before the restart, add a logs source for the same service rather than putting that detail in the alert.

Related