Which labels on Alertmanager alert groups about failed deploys tie it to the right service?

Tares ties a failed-deploy alert to a service through the alert’s own labels, and the label to use is the one every source carries for that service, such as service. The Alertmanager source derives its key from namespace, then service, then job, then instance, or from the label you choose. Mark service as the key on your other sources too, and the alert, metrics, logs and deploys land on one timeline.

How it works

Alertmanager posts a batch of grouped alerts to the source’s ingest URL. Tares splits the alerts array into one event per alert. Each event gets these fields:

• event_type is the alert name, for example a deploy-failure alert name you define.
• text is the alert’s summary annotation.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including its fingerprint, stays in the payload.

The entity key is set automatically from the first of these labels that the alert carries: namespace, service, job, instance. If you want a different label, the advanced key_label field on the source names the alert label to key on. A declared primary label also works.

An entity is a label and value pair, such as service=checkout. A read for that pair returns every event carrying it, across sources, merged in time order. So the failed-deploy alert only joins the right timeline if its label name and value match what your metrics, logs and deploy events carry. Read more about keys and entities in Concepts.

For failed deploys, this means your Prometheus alert rule must put the service identity on the alert as a label. Alertmanager passes that label through, and Tares records it. If the alert only has alertname and severity, there is nothing to correlate on.

Set it up

1. Pick one label name that every source can carry for the same service. service is the usual choice. Make sure the alert rule for failed deploys sets that label, with the same values your other sources use, such as checkout.
2. Create the Alertmanager source in the console under Sources, Add source, Alertmanager alerts, or ask a connected agent to create it and return the webhook URL. Then add the receiver to your alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

send_resolved: true makes the resolved state arrive as events too. If your Tares instance has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. 3. Send a test batch that carries the labels you plan to key on, then check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

4. Open the source’s Fields tab. It shows each alert label with its coverage and top values, so you can see which label is reliably present and promote it, service or namespace, to correlate on. If the default key is not the one you want, set key_label on the source to that alert label.
5. Give the other sources the same key. Mark service as the primary label on the metrics, logs and deploy sources. For deploys, create a webhook source with a primary service label read from the payload’s service field and event_type set to deploy, then have your CD pipeline post one event per deploy.
6. Read the entity: ask a connected agent what happened to checkout in the last 15 minutes. The read uses service=checkout and returns alerts, metrics, logs and deploys together.

The AI SRE guide walks through building this wiring end to end, with a trigger that wakes an agent on the correlated timeline.

What you get

• One timeline per service. A failed-deploy alert sits next to the deploy event, the error logs and the metrics that moved, all on one clock.
• A selector that narrows cleanly. Reading service=checkout returns that service’s evidence, and adding another label narrows it further.
• Resolved states in the same place. With send_resolved on, you see when the alert fired and when it cleared.
• A clear check on label quality. The Fields tab shows how many sampled events carry each label, so you pick a key that is always there rather than one that is sometimes missing.
• Alertmanager keeps its role. It does the grouping, deduping and silencing, and Tares records what it pushes and correlates it with your other sources. See the Alertmanager connector page for the full field reference.

When something else fits

• If you want Tares to poll the alerts that Prometheus rules fire, without Alertmanager’s grouping and silencing, use the prometheus_alerts connector instead.
• If your failed-deploy alerts carry no label that identifies the service, fix that in the alert rule first, because Tares can only correlate on labels the alert has.

Related