Which labels on Alertmanager alert groups about slow database queries tie it to the right service?

Tares ties an Alertmanager alert to a service through one label, and the best choice is a label every source carries for the same service, such as service. Each alert keeps its own labels as Tares labels, so a slow-query alert that carries service lands on the same entity as that service’s metrics, logs and recent changes. If you declare no key, Tares picks one from namespace, then service, then job, then instance.

How it works

Alertmanager POSTs a batch of alerts to the ingest URL of an alertmanager source. Tares fans the batch out into one event per alert. The event type is the alert name, the text is the alert summary, and the alert’s labels become Tares labels, plus a synthetic status of firing or resolved. The full alert, including its fingerprint, stays in the payload.

The entity key comes from one of these, in order:

• A label you mark primary: true on the source.
• The key_label field, an advanced option that names the alert label to key on.
• The automatic choice: namespace, then service, then job, then instance.

An entity is a label and value pair, such as service=checkout. A read for that pair returns the entity’s events from every source that carries the label, merged in time order. See Concepts for how labels, keys and entities fit together.

For slow database queries, this means the label that matters is the one your alert rule attaches. If the rule carries service, the alert joins the service’s timeline. If it carries only instance or job, the alert keys on those and may sit on a different entity than your metrics and logs. Pick one label name and use it everywhere.

Set it up

1. Choose the label. Use one that every source carries for the same service, usually service. Make sure the Alertmanager alert rule for slow queries sets it.
2. Create the Alertmanager source in the console under Sources, Add source, Alertmanager alerts, or ask a connected agent to create it. The source page gives you a receiver block with the ingest URL filled in. Add it to your alertmanager.yml and reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

3. If Tares runs with auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. Without auth, nothing more is needed.
4. Mark the key on the Alertmanager source. Set key_label to the alert label you chose:

- name: db-alerts
  connector: alertmanager
  config:
    key_label: service

5. Mark the same label as the primary label on your other sources, so they all key on one value. For example, a Prometheus source reads it from the metric labels, and a Docker logs source can set it as a constant:

- { name: service, field: metric.service, primary: true }
- { name: service, const: api-server, primary: true }

6. Add recent changes. Create a webhook source with a primary service label read from the payload’s service field, and post one event from your deploy pipeline. Deploys then appear on the same timeline as the alerts.
7. Verify with a test alert, then check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

8. Open the source’s Fields tab. It shows each label’s coverage, meaning how many sampled events carry it, and its top values. Use it to confirm your chosen label is reliably present, and promote it if it is not yet the key. The Alertmanager connector page has the full setup.

What you get

• One entity timeline per service. A read for service=checkout returns the slow-query alert, the metrics, the logs and the latest deploy together on one clock.
• Firing and resolved states for each alert, so you see when a slow-query problem started and when it cleared.
• A reliable key. The coverage numbers on the Fields tab tell you whether the label is present before you rely on it.
• Triggers and agents that work from the same correlated timeline. A trigger can name the label that identifies the entity across its sources, and an agent woken by it reads everything for that service at once. The AI SRE guide walks through this end to end, with every source keyed by the same service label.

When something else fits

• If the alerts come from Prometheus rules and you do not need Alertmanager’s grouping, silencing and resolved semantics, the prometheus_alerts connector polls them instead.
• If the slow-query alert has no service label at all, add one in the alert rule first. Tares keys on the labels the alert carries, so the fix belongs in the rule.

Related