How do I tie alerts from Prometheus to the right service?

In Tares, you tie Prometheus alerts to a service by mapping an alert label, such as labels.service, to the primary label that your metrics and logs sources also carry. Alerts, metrics and logs then merge into one timeline per service. You set the mapping on the prometheus_alerts source in a few lines of YAML.

How it works

The prometheus_alerts connector polls Prometheus’s own alerts, the ones its alerting rules have already fired. Each alert becomes an event. The alert name is the event_type, the summary is the text, and the alert’s labels become Tares labels plus a synthetic state of firing, pending or resolved.

The join happens through labels:

• Each source declares labels with a name, a field (or a const) and, for one label, primary: true.
• The primary label picks the entity an event belongs to. Events from different sources that carry the same label name and value land on the same timeline.
• So if your alerts, your metrics and your logs all carry a label named service, Tares shows them together for each service, on one clock.

For alerts, the value comes from the alert’s own labels, for example labels.service. For Prometheus metrics, it can come from the metric labels, for example metric.service. For container logs, it can be a constant such as api-server. The label name is what matters. Use the same name, service, everywhere.

State is tracked across polls. Each alert fires once, re-emits only when its state changes, and produces a resolved event when it clears, so the service’s timeline shows when the problem started and ended. See the Prometheus alerts connector page for the full configuration.

Set it up

1. Create the source. Ask a connected agent, “Use tares to ingest the firing alerts from the Prometheus at localhost:9090.” Or open Sources → Add source → Prometheus alerts, enter the url and any auth, and click Discover. You can also POST /api/sources with the YAML object below.
2. Let alerts arrive. Discover proposes only the labels every alert is guaranteed to have, severity and state. Per-alert labels such as service or namespace exist only while an alert is firing, so you add them once real alerts have been ingested.
3. Open the source and check the Fields tab. It shows the labels your alerts carry. Pick the one that names the service.
4. Add that field as a label named service and mark it primary:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: service, field: labels.service, primary: true }
      - { name: severity, field: labels.severity }
      - { name: state, field: state }

5. Give your metrics and logs sources a label with the same name. For example, the metrics source reads service from the metric labels, and the logs source sets it as a constant:

# in the prometheus source
labels:
  - { name: service, field: metric.service, primary: true }
 
# in the docker_logs source
labels:
  - { name: service, const: api-server, primary: true }

6. Verify. Recent events on the alerts source shows one event per active alert. Then open Explore, pick a service, and check that its alerts, metrics and logs appear on the same timeline.

The Build an AI SRE guide wires exactly this pattern: three sources keyed by the same service label, one timeline per service.

What you get

• Every alert lands on the service that raised it, next to that service’s metrics and logs.
• A resolved event appears on the timeline when the alert clears, so you see when the problem started and when it ended.
• Triggers can watch the alerts, metrics and logs together, keyed by service, and wake an agent with one correlated timeline instead of three separate views.
• Prometheus keeps owning alerting. Tares ingests what fired, correlates it with the evidence and handles the waking. Nothing is re-thresholded.
• You can filter by severity (blank means all) and turn on include_pending to also see alerts still waiting out their for: duration.

When something else fits

• If your alerts already flow through Alertmanager, point an Alertmanager webhook receiver at a Tares ingest URL and use the alertmanager connector. Map the service label the same way.
• If alerts come from another system as a JSON post, use the webhook connector. You declare a label per payload field, mark one primary, and use Fields to check which payload fields arrive reliably before you pick the key.
• If you want to correlate on something other than the service, such as namespace, make that your primary label and use the same name on every source you want joined.

Related