Which labels on Alertmanager alert groups about error spikes tie it to the right service?

Tares ties an Alertmanager error-spike alert to a service through the alert’s own labels, and you choose one label, such as service, as the key. Every alert label becomes a Tares label, and the key decides which entity timeline the alert lands on. If your metrics, logs and deploy events carry the same label, the alert, the metrics, the logs and recent changes appear on one timeline for that service.

How it works

Alertmanager posts a batch of alerts to the ingest URL of an Alertmanager source. Tares fans the batch out into one event per alert. Each event has these fields:

• event_type is the alert name, for example HighErrorRate.
• text is the alert summary.
• The alert’s labels become Tares labels, plus a synthetic status of firing or resolved.
• The full alert, including its fingerprint, stays in the payload.

The key is the label whose value Tares stores as key_value on the event. For an Alertmanager source Tares derives it automatically from the first of these labels it finds: namespace, then service, then job, then instance. You can also declare a primary label, or set the advanced key_label field to name the alert label to key on.

This matters for error spikes. If your alerts carry both namespace and service, the automatic choice is namespace, so the alert would land on the namespace and not on the service. Setting key_label to service keys it on the service instead.

An entity is a label and value pair, such as service and checkout. A read for one entity returns every event with that label from every source, merged in time order. The alert only joins the timeline of the right service when the other sources use the same label name and value. See Concepts for labels, keys and entities.

Set it up

1. Create the source. Ask a connected agent to create an Alertmanager source and give you the webhook URL, or use Sources → Add source → Alertmanager alerts in the console. The source page shows a receiver block with the ingest URL filled in.

2. Add the receiver to your alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

If your Tares instance has auth on, the receiver also sends a Bearer Authorization header holding an ingest-scoped API key. Without auth on, the header is not needed.

3. Make sure your error-spike alert rules attach the label you want as the key, for example service, so it reaches Tares as an alert label.

4. Open the source’s Fields tab after a few alerts arrive. It shows each label’s coverage (how many sampled events carry it) and its top values. Pick a label that is present on every alert.

5. Set the source’s advanced key_label field to that label, for example service, so it becomes the key instead of the automatic namespace choice.

6. Use the same label name and values on your other sources. In the AI SRE guide, the metrics source, the log source and the alert source are all keyed by service. The deploy webhook source gets a primary service label read from the payload’s service field.

7. Read the result. Ask your connected client for what happened to a service, and it calls read with the selector service=api-server. The response holds the alerts, metrics, logs and deploy events for that service in one time-ordered timeline.

If a trigger watches these sources, its optional key_field names the label that identifies the entity across them. By default it uses the primary label of the first source.

What you get

• An error-spike alert that sits on the same service timeline as the 5xx metrics, the error logs and the deploy that landed minutes earlier.
• Resolved alerts on the same timeline, because send_resolved makes each resolution arrive as an event with status set to resolved.
• One trigger per incident shape, grouped by service, so an agent wakes with a single correlated timeline for the right entity.
• A finding written back to that service’s timeline, so the next investigation starts from the previous conclusion.
• Lookups instead of text scans, because labels are indexed and selecting by service=checkout is a lookup.

When something else fits

• If you do not need Alertmanager’s grouping, silencing and resolved semantics, the prometheus_alerts route in the AI SRE guide polls the alerts that Prometheus rules fire, with a service label you read from the alert’s labels.
• If your alerts carry only namespace, job or instance, you can key on one of those, as long as your other sources use the same label.

Related