Which labels on Alertmanager alert groups about a disk filling up tie it to the right service?

Tares ties a disk-filling alert to a service through one label that every source carries for that service, such as service, which you mark as the key. Alertmanager alert labels become Tares labels, and the key_label setting picks the one to key on. When metrics, logs and deploys use the same label, the alert and its evidence land on one entity timeline.

How it works

Alertmanager sends a batch of alerts to your Alertmanager source, and Tares turns each alert in the batch into its own event. The event type is the alert name, the text is the alert summary, and every alert label becomes a Tares label. Tares adds a status label that is firing or resolved. The full alert, including its fingerprint, stays in the payload.

The key is the label an event is filed under. For an Alertmanager source Tares derives it automatically from the first label present in this order: namespace, then service, then job, then instance. You can override that with the key_label setting, which names the alert label to key on.

This matters for disk alerts. A disk-filling alert often carries labels such as namespace or instance. If it carries namespace, the default key is the namespace, not the service. If you want the alert on the service’s timeline, set key_label to the label that names the service.

An entity is a label and value pair, such as service and checkout. A read of that entity returns events from every source that carries the same label and value, merged in time order. Read more in Concepts.

Set it up

1. Pick the label every source can carry for the same service. The usual choice is service. Check your alert rules so that the disk-filling alert includes that label.
2. Create the Alertmanager source in the console under Sources → Add source → Alertmanager alerts, or ask a connected agent to create it. Tares gives you an ingest URL and a copy-paste receiver block. See the Alertmanager connector for the full receiver config.
3. Set key_label on the source so the alert is keyed on your service label:

- name: alerts
  connector: alertmanager
  config:
    key_label: service

4. Point an Alertmanager receiver at the ingest URL with send_resolved: true, then reload Alertmanager. When Tares has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key.
5. Send a test batch and check Recent events:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

6. Open the source’s Fields tab. It shows the alert labels, so you can confirm the service label is present and promote namespace or service to correlate on. Coverage figures there show how many sampled events actually carry each label, which helps you choose a key that is reliably present.
7. Key your other sources on the same label. Your Prometheus metrics source reads the service from the metric labels, your logs source (Docker logs or OTLP) carries it per container or per resource, and a webhook source for deploys and config changes gets a primary service label read from the payload’s service field. The AI SRE guide walks through wiring all of these to one service label.

What you get

• A disk-filling alert that lands on the same entity timeline as that service’s metrics, logs and recent deploys.
• One read, such as service equal to checkout, that returns all of that evidence merged on one clock, so you can see a deploy, the growing disk metric and the alert in order.
• A firing and resolved status on the alert events, so the timeline shows when the disk problem started and when it cleared.
• A key you chose on purpose, instead of whichever of namespace, service, job or instance the alert happened to carry first.
• Triggers and agents that work per service. A trigger keyed on the same label wakes an agent with the service’s timeline, and the agent can write its finding back onto it.

When something else fits

• If the alerts in your Alertmanager do not carry any label that names the service, add one in your alert rules or routing first. Keying on instance or job still works, but then the timeline is per instance or per job.
• If you want Prometheus to own alert evaluation and Tares to poll what fired, the prometheus_alerts connector fits better, though you give up Alertmanager’s grouping, silencing and real-time delivery.

Related