Which labels on Alertmanager alert groups about high latency tie it to the right service?

Tares ties a high latency alert to a service through a label every source carries for that service, such as service, which you mark as the key. The alertmanager source turns each alert in a group into one event and keeps the alert’s labels as Tares labels. Metrics, logs, deploys and the alert then land on one entity timeline for that service.

How it works

Alertmanager posts a batch of alerts to the source’s ingest URL. Tares fans the alerts array out into one event per alert. Each event has event_type set to the alert name, text set to the alert summary, the alert’s labels as Tares labels, and a synthetic status of firing or resolved. The full alert, including its fingerprint, stays in the payload.

The key decides which entity an event belongs to. The alertmanager source derives it automatically from the first of these labels it finds: namespace, then service, then job, then instance. You can also declare a primary label, or set the advanced key_label field to the alert label you want to key on.

This order matters for latency alerts. If your alerts carry both namespace and service, the automatic choice is namespace. Set key_label to service when you want the timeline per service rather than per namespace.

A read on an entity such as service = checkout returns every event with that label from every source, merged in time order. The selector is a strict AND, so adding a label narrows the result. For the full model, see Concepts.

Set it up

1. Make sure each high latency alert carries a service label with the same value your metrics and logs use for that service.
2. In the console, go to Sources, then Add source, then Alertmanager alerts. Name the source and click Create. The source page shows a copy-paste receiver block with the ingest URL filled in.
3. Add the receiver to your alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

If Tares runs with auth on (tares up --auth), the Alertmanager sender uses a Bearer Authorization header holding an ingest-scoped API key. Without auth on, you need no key. The source page shows the auth block when the instance requires a token. 4. Send a test batch, then check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

5. Open the source’s Fields tab. It shows each alert label with its coverage, meaning how many sampled events carry it, and its top values. Confirm service is present on your latency alerts and promote it to the correlation label.
6. Set the advanced key_label field to service so the key is the service and not the namespace.
7. Give the other sources the same label. In the AI SRE guide, the metrics, logs and alerts sources are all keyed by service. The metrics source reads service from the metric labels, and a deploy webhook source has a primary service label read from the payload’s service field.

For the receiver details and the key_label field, see the Alertmanager connector page.

What you get

• One timeline per service. A high latency alert for checkout sits next to the latency metric, the request logs and the latest deploy for checkout, all on one clock.
• Firing and resolved events on the same timeline, because send_resolved: true makes Alertmanager send resolved alerts too. You see when latency started and when it recovered.
• Triggers keyed by service. A trigger over these sources emits one dispatch per service, carrying that service’s timeline, so the agent it wakes starts from the correlated evidence.
• Alertmanager keeps its grouping, silencing and dedup. Tares records what Alertmanager pushes and correlates it with your other sources.
• A coverage check before you commit. The Fields tab tells you whether service is on most of your alerts, so you pick a key that is reliably present.

When something else fits

• If you want Tares to read alerts straight from Prometheus rules, without Alertmanager’s grouping and silencing, use the prometheus_alerts connector, which polls every 30 seconds by default. The Alertmanager connector is the better fit for real-time delivery and resolved semantics.
• If your latency alerts carry no service label, the automatic key falls back to namespace, job or instance. Add a service label in your alert rules, or pick another label that all your sources share, and key on that.

Related