What should Alertmanager alert groups about 5xx errors on an API include for an AI SRE?

Tares records every alert in an Alertmanager group as its own event, so each 5xx alert should carry a service label, a severity label, a summary annotation and a start time. Tares uses the alert name as the event type, the summary as the event text and the labels to key the event to a service. With those four fields the agent can tie the alert to the metrics and logs for the same service.

How it works

Alertmanager POSTs a batch to your Tares ingest URL as { alerts: [...], ... }. Tares fans the alerts array out into one event per alert:

• event_type is the alert name, so name the alert for the symptom, for example HighErrorRate.
• text is the alert’s summary annotation, so the summary is the sentence the agent reads first.
• The alert’s labels become Tares labels, and Tares adds a status of firing or resolved.
• The full alert, including fingerprint and the start time, stays in the event payload.

Tares derives the entity key from a label, checking namespace, then service, then job, then instance. You can declare a primary label instead, or set the advanced key_label field on the source. When your metric and log sources carry the same service label, the AI SRE guide shows how Tares merges them into one timeline per service, and the alert lands on that timeline next to the evidence.

So for a 5xx alert, put these on every alert:

• A service label, such as service set to api-server, using the same value your metric and log sources use. It decides which timeline the alert joins.
• A namespace label, if you use one. It is checked first when Tares picks the key.
• A severity label, such as critical or warning, so the agent and your triggers can tell a page from a notice.
• A summary annotation, a plain sentence such as 5xx rate is high on api-server. It becomes the event text.
• A start time, startsAt, which Alertmanager sends with each alert and Tares keeps in the payload.

Also set send_resolved: true on the receiver. Tares then gets a resolved event when the 5xx rate recovers, so the timeline shows when the problem started and when it ended.

Set it up

1. Create the source. In the console go to Sources → Add source → Alertmanager alerts, name it and click Create. Or ask a connected agent: “Use tares to create an Alertmanager source and give me the webhook URL.” Either way you get an ingest URL.
2. Add a receiver to your alertmanager.yml that points at that URL, then reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If your Tares runs with auth on (tares up --auth), the sender uses a Bearer Authorization header holding an ingest-scoped API key. The source page shows the auth block already filled in. Without auth you need nothing extra.
4. Send a test batch shaped like a good 5xx alert, with the service, severity, summary and start time all present:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"HighErrorRate","severity":"critical","service":"api-server"},"annotations":{"summary":"5xx rate is high on api-server"},"startsAt":"2024-01-01T00:00:00Z"}]}'

5. Open Recent events on the source and check that the event appears with the alert name as its type and your summary as its text. Open Fields to see the alert labels, and promote service (or namespace) so Tares correlates on it.

The Alertmanager connector page has the full reference for the receiver and the key_label field.

What you get

• One event per 5xx alert, so a group of alerts does not collapse into a single opaque message.
• An event type that names the symptom and a text line the agent can quote.
• A firing and resolved status on each event, so the agent sees the start and the recovery.
• Alerts keyed by service, so they sit on the same timeline as your metrics and logs. The agent can tie HighErrorRate on api-server to the error log lines and the 5xx metric from the same minutes.
• A severity label you can filter on in triggers.
• The full alert, fingerprint included, kept in the payload for the agent to read.

Alertmanager keeps owning grouping, silencing and deduplication. Tares records what it pushes and correlates it with your other sources.

When something else fits

• If you do not run Alertmanager and want Prometheus rules polled directly, the prometheus_alerts connector reads fired alerts every 30 seconds by default. Alertmanager gives you grouping, silencing and real-time delivery, so use it when you already depend on those.
• If your alert labels use no service name, declare a primary label or set key_label on the source, so the alert still joins the right timeline.

Related