Which labels on alerts from Prometheus about error spikes tie it to the right service?

Tares ties an error-spike alert to its service through a service label that you mark as the key (primary: true) on the alerts source. Use the same label name on your metrics, logs and deploy sources, so the alert and its evidence land on one timeline for that service. Prometheus alerts always carry severity and state, and the service label comes from your alerting rules.

How it works

Every event carries labels, and the label marked primary: true is the key. An entity is a label and value pair, such as service and checkout. Events from different sources that share that pair merge into one time-ordered timeline. See Concepts for the full model.

For the Prometheus alerts connector, these points matter:

• Each active alert becomes an event. Its event_type is the alert name, such as an error-rate alert, and its text is the summary.
• The alert’s own Prometheus labels become Tares labels, plus a synthetic state of firing, pending or resolved.
• Setup proposes only the labels guaranteed to be present, severity and state. Per-alert labels such as namespace and service exist only while an alert is firing, so you promote them after real alerts have been ingested.
• If no label is marked primary, the connector’s default key is used. That default is not necessarily the service you want, so set the key yourself.

The label to pick is one that every source carries for the same service. The Fields tab of a source reports each label’s coverage (how many sampled events carry it) and its top values, so you can confirm the label is reliably present before you make it the key.

Set it up

1. Create the alerts source from Sources, Add source, Prometheus alerts, or ask a connected agent. The starting config proposed for you has severity as the key:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

2. Let an error spike fire, then open the source’s Fields tab. Check that service appears with high coverage.
3. Move the primary: true flag from severity to service, and keep severity and state as plain labels. The final block looks like this:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: service, field: labels.service, primary: true }
      - { name: severity, field: labels.severity }
      - { name: state, field: state }

4. Give your other sources the same service label as key. For metrics from Prometheus, read it from the metric labels:

labels:
  - { name: service, field: metric.service, primary: true }

5. For container logs, set it as a constant:

labels:
  - { name: service, const: api-server, primary: true }

6. For recent changes, create a webhook source for deploys with a primary service label read from the payload’s service field. Your CD pipeline posts one event per deploy.
7. Open Explore, pick the service, and read the merged timeline.

The Build an AI SRE guide walks through this wiring with a demo system, where all three sources are keyed by the same service label.

What you get

• One timeline per service. The error-spike alert, the metric that tripped it, the error logs and the latest deploy sit on one clock under the same entity.
• Alerts that fire once and emit a resolved event when they clear, so the timeline shows when the spike started and ended.
• A trigger keyed by service that fires per service, so the agent it wakes gets that service’s correlated evidence and not a pile of unrelated alerts.
• Findings written back to the same entity, so the next investigation starts from the previous conclusion.
• Indexed labels, so selecting events for one service is a lookup and not a text scan.

When something else fits

• If your alerts route through Alertmanager and you want to keep that path, the alertmanager connector option in the AI SRE guide ingests what Alertmanager fires, and you key it by service the same way.
• If your alerting rules do not attach a service label to the alert, pick the label they do carry, such as namespace, and use it on every source instead. Consistency across sources matters more than the name.

Related