Which labels on Prometheus alerts about a disk filling up tie it to the right service?

Tares ties a disk-filling alert to its service through a shared label such as service, which you mark as the key so alerts, metrics, logs and changes land on one timeline. The alert’s own Prometheus labels become Tares labels on each event. Declare the same label name on every source and the alert joins that service’s entity.

How it works

Tares reads each active alert from Prometheus with the prometheus_alerts connector (see Prometheus alerts). Every alert becomes one event. The event_type is the alert name, the text is its summary, and the alert’s own labels become Tares labels. Tares adds a state label that is firing, pending or resolved.

The label that ties the alert to a service is whichever one you declare on the source and mark primary: true. That label is the key, and the event’s key_value is set from it. An entity is a (label, value) pair, such as service=api-server. Every source that carries service with the same values lands on the same entity. This is how a concept page describes it: the labels on an event say which entity it belongs to, and a read returns that entity’s events across sources in time order.

For a disk filling up, this gives you:

• The alert from prometheus_alerts, with its service label read from the alert’s labels.
• The metrics from the prometheus connector, with service read from the metric labels.
• The logs from docker_logs or OTLP, with service as a constant or read from the event.
• Recent changes from a webhook source, with service read from the payload.

When the alert clears, Tares emits a resolved event with the same key, so the timeline shows when the disk problem started and when it ended.

Set it up

1. Pick one label name that every source can carry for the same service. service is the usual choice. Use the same spelling and the same values everywhere, so that api-server in the alert matches api-server in the logs.

2. Add the alerts source under Sources → Add source → Prometheus alerts. Enter the Prometheus url and run Discover. Discover proposes only the labels guaranteed on every alert, which are severity and state. Labels such as namespace and service exist only while an alert is firing, so they are not proposed until alerts have been ingested.

3. Let the disk alert fire once, then open the source’s Fields tab. It shows which labels the alerts actually carry, how many sampled events have each one (coverage) and their top values. Choose the label with reliable coverage that names your service, and promote it. Add namespace too if your alerts carry it.

4. Mark the service label as the key. The source below uses the labels format from the connector page, with service as primary:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: service, field: labels.service, primary: true }
      - { name: severity, field: labels.severity }
      - { name: state, field: state }

5. Give your other sources the same label. The metrics source reads service from the metric labels, for example field: metric.service. The logs source for a container can use a constant, such as { name: service, const: api-server, primary: true }. A webhook source for deploys and config changes reads service from the payload’s service field.

6. Start Tares if it is not running, then open Explore and pick the service.

tares up

If no label is marked primary, the connector’s default key is used. Set the primary label yourself so the key is the one every source shares. The AI SRE guide walks through this layout end to end, with three sources all keyed by service.

What you get

• One timeline per service. In Explore, pick service=api-server and read the disk alert, the filesystem metrics, the container logs and the latest deploy merged on one clock. Selecting by label is an indexed lookup, not a text scan.
• Context beside the alert. The warning and critical alerts, their state changes and the resolved event sit next to the metrics that tripped them and the change that may have caused them.
• Triggers that use the same key. A trigger’s key_field defaults to the primary label of the first source it watches, so a firing groups per service and carries that service’s timeline to the agent it wakes.
• Findings in the right place. An agent that diagnoses the disk alert writes its finding onto the same service entity, so the next investigation starts from that conclusion.
• One firing per state change. Tares tracks alert state across polls, so a long-running disk alert is not repeated every tick.

When something else fits

• If your alerts reach Tares through Alertmanager, use the alertmanager connector instead. The labeling idea is the same: declare service on the source and mark it primary.
• If your alerts carry no label that names a service, add a constant label on the alerts source, or key on another label that every source shares, such as namespace. Choose the one with the best coverage on the Fields tab.

Related