How do I line up alerts from Prometheus with OpenTelemetry metric series on one timeline?

Tares puts Prometheus alerts and OpenTelemetry metric points on one timeline when both sources carry the same service label. Add a prometheus_alerts source and an otlp source, both keyed by service, then open Explore and pick the service. You see the alert that fired and the metric points that led up to it on one clock.

How it works

Tares joins events on a shared key. Here the key is the service label, and each source fills it in its own way:

• Alerts. The Prometheus alerts connector polls Prometheus’s /api/v1/alerts every 30 seconds by default. Each active alert becomes one event. The event_type is the alert name, the text is its summary, and the alert’s labels become Tares labels, plus a state of firing, pending or resolved. Tares tracks state across polls, so an alert fires once and a resolved event follows when it clears.
• Metrics. The OpenTelemetry connector receives OTLP/HTTP logs, traces and metrics. Resource attributes such as service.name become labels, so the service label comes straight from your SDK or Collector. One source ingests every service.

Because both carry service, the events for one service merge into one timeline. This is the same per-service read that the AI SRE guide builds its agent on. Prometheus keeps deciding what counts as an alert, and Tares correlates what fired with the metric points around it.

Set it up

1. Start Tares.

   tares up

2. Add the OpenTelemetry source under Sources → Add source → otlp, or ask a connected agent to create an OTLP source keyed by service. There is no required config, only the service label.

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: service, primary: true }

3. Point your exporter at Tares. Use the OTLP/HTTP endpoint, or the gRPC endpoint if you installed the otlp-grpc extra.

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

   The sender uses a Bearer Authorization header holding an ingest-scoped API key, needed only when Tares has auth on.

4. Add the alerts source under Sources → Add source → Prometheus alerts. Enter the Prometheus url, run Discover, and optionally filter by severity.

   - name: alerts
     connector: prometheus_alerts
     poll: 30s
     config:
       url: http://localhost:9090
       severities: critical,warning   # blank = all
       labels:
         - { name: severity, field: labels.severity, primary: true }
         - { name: state, field: state }

5. Give the alerts a service label. Discover proposes only the labels every alert is guaranteed to have, which are severity and state. An alert’s own labels, such as service or namespace, exist only while it fires. Let a real alert arrive, open the source’s Fields tab, and add a service label from the field that carries it. Use the same label name as on the OTLP source.

6. Check both sources. Recent events on the OTLP source shows metric points as they arrive. Recent events on the alerts source shows one event per active alert. Then open Explore, pick the service under the service key, and read the alert and the metric points together.

What you get

• One timeline per service, with the alert event and the OpenTelemetry metric points merged on one clock.
• The cause and the effect side by side. The alert shows what fired and when. The metric points show how the service got there.
• A resolved event on the same timeline when the alert clears, so you can see the metrics recover after it.
• A read that works for a person in Explore and for an agent. An agent subscribed to a trigger gets the same correlated timeline when an alert event arrives.
• No extra setup per service. The OTLP source covers every service your exporters report, and each service shows up under the service key.

When something else fits

• If you want Prometheus-scraped metrics rather than OTLP ones on the timeline, use the Prometheus connector. You pick metrics by name or label with no PromQL, and the value is stored as a number you can aggregate in triggers.
• If your alerts reach you through Alertmanager rather than Prometheus’s own rules, use the alertmanager connector. It joins the same service timeline the same way.

Related