How do I line up Prometheus alerts with OpenTelemetry metric series on one timeline?
Tares lines up Prometheus alerts and OpenTelemetry metric series on one clock by giving both sources the same primary service label. Add a prometheus_alerts source and an otlp source, and Tares merges what fired and how the metric moved into one timeline per service. You read it in Explore or through a connected agent, as described in Build an AI SRE.
How it works
Tares joins events on a shared key. When two sources carry the same primary label, their events land on the same entity’s timeline, ordered by time.
- The Prometheus alerts connector polls Prometheus’s
/api/v1/alerts. Each active alert becomes one event. Theevent_typeis the alert name,textis its summary, and a syntheticstatelabel isfiring,pendingorresolved. State is tracked across polls, so an alert fires once and aresolvedevent follows when it clears. - The OpenTelemetry connector receives OTLP/HTTP metrics (and logs and traces) at
POST /v1/metrics. One source ingests every service, and the metric points show up as events. - The shared key is
service. On the OTLP source it comes from the resource attributeservice.name, so the label readsresourceAttributes.service.name. OTLP events have no bareservicefield, so a label written asfield: servicegets no value there. - On the alerts source,
serviceis an alert label. It is not guaranteed to exist, because an alert’s dynamic labels only exist while it is firing.
Mark the service label primary on both sources. The alert’s service value must match the OTel service.name for the same service, or the two will land on different timelines.
Set it up
-
Start Tares and open the console.
tares up -
Add the OTLP source under Sources, Add source, otlp, or ask a connected agent to create it. It has no required config, only the label.
- name: otel connector: otlp config: labels: - { name: service, field: resourceAttributes.service.name, primary: true } -
Point your OTel SDK or Collector at Tares. The endpoint below is the OTLP/HTTP one. For gRPC you install the
otlp-grpcextra and use port 4317.OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787 # OTLP/HTTP → /v1/{logs,traces,metrics} # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317The exporter sends an
Authorization: Bearerheader holding an ingest-scoped API key, needed only when Tares has auth on. -
Add the alerts source under Sources, Add source, Prometheus alerts. Enter the Prometheus
urland press Discover. Discover proposesseverityandstate. Once real alerts have been ingested, promote the service label from the Fields tab and mark it primary, so the source looks like this:- name: alerts connector: prometheus_alerts poll: 30s config: url: http://localhost:9090 severities: critical,warning # blank = all labels: - { name: service, field: labels.service, primary: true } - { name: severity, field: labels.severity } - { name: state, field: state } -
Check both sides. On each source, Recent events shows what is arriving. On the OTLP source it lists metric points as they come in. In Explore, pick the service under the
servicekey and read the alerts and metric points together.
If you also want metrics that Prometheus itself scrapes, the Prometheus connector ingests the metrics you pick by name or label, with no PromQL, and stores the value as a number-typed value label. Give it a primary service label too and it joins the same timeline.
What you get
- One timeline per service, with the alert and the metric points on the same clock.
- The alert tells you what fired and when. Its
stateshows the move fromfiringtoresolved, and its text carries the alert’s summary. - The OTel metric points sit next to it, so you can see how the series moved around the moment the alert fired and when it cleared.
- Each alert is one event per state change, not a copy on every poll.
- The same read is available to a connected agent, and the Explore view shows exactly what an agent gets. When a trigger fires on an alert event, the agent wakes with this correlated timeline. The AI SRE guide walks through that end to end.
- Prometheus keeps owning alerting. Tares ingests what fired and correlates it, and nothing is re-thresholded.
When something else fits
- If you already route alerts through Alertmanager, use the
alertmanagerconnector instead of polling. The sameservicekey still joins it with OTLP. - If the metrics you want are already in Prometheus, the Prometheus connector alone may be enough, and you can skip OTLP for those series.
- If your alerts do not carry a service label at all, promote whichever label identifies the service from the Fields tab first. Without a shared key, the two sources stay on separate timelines.