How do I pair alerts from Prometheus with OpenTelemetry log records for an AI SRE?

Tares pairs the two by giving the prometheus_alerts source and the otlp source the same service label. Every firing alert then lands on one per-service timeline next to the log records from the same period. An AI SRE agent reads that single timeline, so it starts from the alert and the evidence together.

How it works

Tares correlates everything that carries the same label value. You keep two sources and give both a service label.

• The Prometheus alerts connector (prometheus_alerts) polls Prometheus’s own alerts endpoint every 30 seconds. Prometheus decides what counts as an alert. Tares ingests what fired.
• Each alert becomes one event. Its event_type is the alert name and its text is the summary. The alert labels become Tares labels, plus a synthetic state of firing, pending or resolved.
• Tares tracks state across polls, so an alert fires once and emits a resolved event when it clears, not a copy on every poll.
• The OpenTelemetry connector (otlp) receives OTLP logs, traces and metrics. One source takes every service, and resource attributes such as service.name become labels. The service label comes from service.name.
• Because both sources carry service, Tares merges alerts and log records for that service onto one timeline on one clock. A trigger keyed by service can wake an agent when an alert event arrives, and the agent reads the correlated timeline instead of querying each system.

One detail matters for the alert side. An alert’s dynamic labels, such as namespace and service, exist only while the alert is firing. Discovery proposes only severity and state. You add service after real alerts have been ingested.

Set it up

1. Start Tares and open the console.

   tares up

2. Add the alerts source under Sources → Add source → Prometheus alerts. Enter the Prometheus url and run Discover. Optionally filter by severity and toggle include pending. Save it with the default labels.

3. Let an alert fire, then open the source’s Fields tab. The labels the alerts carry are listed there. Promote service to a label and make it the primary one.

   - name: alerts
     connector: prometheus_alerts
     poll: 30s
     config:
       url: http://localhost:9090
       severities: critical,warning   # blank = all
       labels:
         - { name: service, field: labels.service, primary: true }
         - { name: severity, field: labels.severity }
         - { name: state, field: state }

4. Add the log source under Sources → Add source → otlp. No connector-specific config is required. Use the service label as the key.

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: service, primary: true }

5. Point your exporter, an instrumented app or an OpenTelemetry Collector, at Tares.

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

   OTLP/gRPC needs the otlp-grpc extra and listens on port 4317. If Tares has auth on, the exporter sends an ingest-scoped API key in a Bearer Authorization header. Without auth you need no header.

6. Make sure the service value in the alert labels matches the service.name your apps export. Correlation only works when the two values are identical, for example api-server on both sides.

7. Check the result. Recent events on each source shows events as they arrive. Explore lists the service under the service key, and opening it shows alerts and log records on one clock.

8. Add a trigger over both sources, keyed by service, that fires when an alert event arrives. Then pick who it wakes: a Tares agent, your own agent service over a webhook, or a Slack channel. The AI SRE guide walks through this whole setup, including the trigger and the agent.

What you get

• One timeline per service, with each alert and its log records side by side on the same clock.
• A resolved event on that timeline when the alert clears, so the agent sees the whole life of the incident.
• A single read for the agent. A woken agent, or a connected MCP client, gets the correlated evidence for the service without querying Prometheus and your log store separately.
• A firing alert that wakes the agent with its logs already attached.
• Findings the agent writes back onto the same timeline, so the next investigation starts from the previous conclusion.
• No change to your alerting. Prometheus keeps owning the rules, and Tares ingests what fires and correlates it.

When something else fits

• If you run Alertmanager and want its routing and grouping to decide what reaches Tares, use the alertmanager connector instead of polling Prometheus. The guide lists it as the alternative for alerts, and you keep the same service label.
• If your logs come from a single container rather than an OpenTelemetry pipeline, the docker_logs connector is the simpler source. Key it by the same service label.

Related