Which labels on Prometheus alerts about slow database queries tie them to the right service?

Tares ties a slow-query alert to the right service through one label that every source carries, usually service, which you mark as the key. Give the alert source, the metrics, the logs and your deploy events that same label. They then land on one entity timeline for that service. Prometheus keeps deciding what counts as an alert, and Tares correlates it.

How it works

Every event in Tares carries labels, and the label marked primary: true is the key. Its value is stored as key_value. An entity is a label and value pair, such as service=checkout. A read for that pair returns the service’s events from every source, merged in time order.

For the Prometheus alerts connector, each active alert becomes one event:

• event_type is the alert name, for example your slow-query rule.
• text is the alert summary.
• The alert’s own labels become Tares labels, plus a synthetic state of firing, pending or resolved.
• A resolved event is emitted when the alert disappears.

Two labels are always present on this source: severity and state. They describe the alert, not the service, so they make poor keys. The alert’s dynamic labels, such as namespace or service, exist only while the alert is firing. That is why you add them after real alerts have been ingested, not at the start.

The key must be the same label name on every source you want on the timeline. A source’s Fields tab reports each label’s coverage, meaning how many sampled events actually carry it, and its top values. Use that coverage figure to pick a key that is reliably present.

Set it up

1. Create the alert source. In the console go to Sources, Add source, Prometheus alerts, enter the Prometheus url and click Discover. You can also ask a connected agent to ingest the firing alerts. Optionally filter by severity.
2. Start with the labels Discover proposes. This is the shape the connector page shows:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

3. Let a slow-query alert fire, then open the source’s Fields tab. Look for the label that names the service on your alert, such as service. Check its coverage and top values.
4. Add that label to the source and mark it primary: true, so it becomes the key. Take the field name from what the Fields tab shows for your alerts.
5. Use the same label name as the key on the other sources: your Prometheus metrics, your container or OpenTelemetry logs, and a webhook source for deploys. In the AI SRE guide all of them are keyed by service.
6. Open Explore, pick the service, and confirm that the alert, metrics, logs and deploys appear on one clock. A trigger over these sources uses the primary label of its first source as the entity unless you set key_field.

The concepts page defines labels, keys and entities in full, if you want the background.

What you get

• One timeline per service. A slow-query alert sits next to the latency and connection metrics, the database errors in the logs, and the deploy that landed just before it.
• Lookups instead of text scans. Labels are indexed, so a read for service=checkout is a direct lookup, and adding a second label such as namespace narrows it.
• Triggers that group by the same entity. A firing carries that service’s timeline across the trigger’s sources, and an agent woken by it reads one correlated view.
• Findings on the right entity. An agent’s diagnosis is written back with the same label as the evidence, so the next investigation of that service starts from it.

When something else fits

• If your alerts carry no label that names a service, there is nothing for the key to match on. Fix the label in your Prometheus rules first, or choose another label every source shares, such as namespace.
• If you need a single stable label before any alert has fired, use a const label on the source. Switch to the alert’s own label once real alerts show up in the Fields tab.

Related