Which labels on Alertmanager alert groups about crash loops tie it to the right service?

Tares ties a crash-loop alert to its service through the entity key, and the label to use is the one every source carries for that service, such as service. Mark it as the key on each source, and on the Alertmanager source set key_label to service. The alert, the metrics, the logs and recent deploys then land on one entity timeline.

How it works

Each alert Alertmanager sends becomes one Tares event. Its event_type is the alert name, its text is the summary, and the alert’s own labels become Tares labels, plus a synthetic status of firing or resolved. The full alert, including fingerprint, stays in the payload.

The key decides which entity the event belongs to. Without a setting, the Alertmanager source derives it from the first of these labels the alert carries: namespace, then service, then job, then instance. A crash-loop alert that carries namespace is therefore keyed by namespace, not by service, unless you say otherwise.

Two settings fix that:

• key_label on the Alertmanager source names the alert label to key on.
• A label marked primary: true on any source sets that event’s key_value.

An entity is a label and value pair, for example (service, checkout). A read for { service: "checkout" } returns every event with that label, merged in time order. For that to include the crash-loop alert, the metrics, the logs and your deploys, each of those sources must carry the same label name with the same values. See Concepts for how labels, keys and entities fit together.

Set it up

1. Pick the label every source can carry for the same service. service is the usual choice. Make sure your crash-loop alert rule attaches it, and that the values match across sources (for example checkout everywhere).

2. Create the Alertmanager source. Ask a connected agent to create an Alertmanager source, or use Sources, Add source, Alertmanager alerts in the console. Then add the receiver to your alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

If Tares has auth on, the receiver also sends a Bearer Authorization header holding an ingest-scoped API key. Without auth, no header is needed. Details are on the Alertmanager connector page.

3. Tell the source to key on that label:

- name: alerts
  connector: alertmanager
  config:
    key_label: service

4. Give the other sources the same label and mark it as the key. This is the declaration shape from the concepts page:

labels:
  - { name: service, field: service, primary: true }
  - { name: env, const: prod }

Use a field label where each event carries the service, and a const label where one source covers one service. Metrics from Prometheus, logs from Docker or OTLP, and a webhook source for deploys all follow this pattern. The AI SRE guide shows all of them keyed by service.

5. Send a test alert and check the result:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

6. Open the source’s Fields tab. It reports how many sampled events carry each label and its top values, so you can confirm service is reliably present before you rely on it. Then open Explore, pick the service, and read the merged timeline.

If you add a trigger over these sources, its optional key_field names the label that identifies the entity across them. Without it, the trigger uses the primary label of its first source.

What you get

• A crash-loop alert that lands on the service’s timeline, not only on a namespace or instance.
• One read that returns the alert, the metrics, the logs and recent deploys merged on one clock, so an agent or a person sees what changed before the restarts began.
• Firing and resolved events for the same alert on the same entity, because send_resolved is on and status is recorded on each event.
• Triggers that group by the same entity, so one dispatch carries the service’s whole timeline across the sources the trigger watches.
• Findings that agents write back onto the same entity, so the next investigation of that service starts from the earlier conclusion.
• The original alert preserved unmodified in the payload, including its fingerprint.

When something else fits

• If your alerts already carry namespace and one namespace maps to one service, the default key derivation works and you can skip key_label.
• If you want Tares to poll Prometheus rules directly instead of receiving Alertmanager’s grouped and silenced alerts, use the prometheus_alerts connector, which polls every 30 seconds by default.
• If your alert rules cannot attach a service label, add a const label on a source that covers a single service, or fix the rule so every source agrees on the label.

Related