Which labels on alerts from Prometheus about failed deploys tie it to the right service?

Tares ties a Prometheus alert to the right service through the service label the alert carries, which you mark as the key (primary) on the source. Use the same service label name on your metrics, logs and deploy sources, and the alert, the evidence and recent changes land on one entity timeline. The severity and state labels are always present, but they describe the alert, not the service.

How it works

The prometheus_alerts source polls Prometheus’s own alerts endpoint. Each alert becomes an event with event_type set to the alert name, text set to its summary, and the alert’s labels as Tares labels, plus a state label of firing, pending or resolved.

Two kinds of label matter here:

• Always-present labels. severity and state are on every alert, so Discover proposes them. They group alerts by how bad and how current they are. They do not say which service failed.
• Per-alert labels. Labels such as namespace and service exist only while an alert is firing, so Discover does not propose them. They come from your alerting rules, so the rule that catches a failed deploy needs to put the service on the alert.

The label you mark primary: true is the key, and each event’s key_value is set from it. An entity is a pair of label and value, for example service and checkout. A read for that pair returns every event with that label, merged in time order. For one read to include the alert, the metrics, the logs and the deploy events, every source must carry the same label name with the same values. The Concepts page defines labels, keys and entities in full.

The AI SRE guide uses this pattern. Its metrics, logs and alerts sources are all keyed by service, so opening api-server in Explore shows them on one clock.

Set it up

1. Create the alerts source from Sources, Add source, Prometheus alerts, or ask a connected agent to ingest the firing alerts from your Prometheus. Enter the url, add any auth, and click Discover. This starting config comes from the Prometheus alerts connector page:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

2. Let the failed-deploy alert fire once, so Tares has a real alert to read. The service label does not exist until then.
3. Open the source’s Fields tab. It lists the labels the ingested alerts carry, with coverage and top values. Check that service is present on the alerts you care about, and promote it to a label of the source.
4. Mark service as the key, primary: true, in place of severity. Pick a label that shows high coverage on the Fields tab, so the key is reliably present.
5. Give the other sources the same service label name. For metrics, read service from the metric labels on the Prometheus connector. For a container’s logs, use a constant such as { name: service, const: api-server, primary: true }. For deploys, create a webhook source with a primary service label read from the payload’s service field and event_type: deploy, then post one event per deploy from your CD pipeline.
6. Open Explore, pick the service, and confirm the alert, the metrics, the logs and the deploy events appear together.

If you start Tares locally, tares up brings up the console:

tares up

What you get

• One timeline per service, with the failed-deploy alert, its metrics, its logs and the deploy event that preceded it on one clock.
• Alerts that fire once and emit a resolved event when they clear, so the timeline shows when the problem started and ended.
• Triggers keyed by service, so a firing is one dispatch for one service, carrying that service’s timeline across the sources the trigger watches.
• Findings from agents written back to the same entity, so the next investigation starts from the previous conclusion.
• A key that is a lookup, not a text scan, because labels are indexed.

When something else fits

• If your alerts reach you through Alertmanager, use the alertmanager connector instead. It makes one event per alert and carries the same kind of labels.
• If the alert carries no service label at all, add one in your alerting rule first, or key on another label that every source shares, such as namespace, so the sources still meet on one entity.

Related