Which OpenTelemetry log records does an AI SRE search when alerts from Prometheus fire?

Tares gives an AI SRE the OpenTelemetry log records of the service that the Prometheus alert is about, not every log you ingest. Both signals carry the same service label, so they merge into one correlated timeline per service. When an alert fires, the agent reads that service’s timeline, with the alert event sitting next to the logs.

How it works

The narrowing comes from one shared label, not from a search query you write.

• OTLP logs get a service label. The OpenTelemetry connector receives OTLP/HTTP logs, traces and metrics. Resource attributes such as service.name become labels, so the service label identifies which service each log record came from. One source ingests every service, with no per-service setup.
• Alerts get the same label. The Prometheus alerts connector polls Prometheus’s own alerts and creates one event per active alert, plus a resolved event when it clears. Each alert becomes an event with event_type set to the alert name, text set to its summary, and its labels as Tares labels plus a state of firing, pending or resolved.
• One timeline per service. Because both sources carry service, metrics, logs and alerts for api-server are merged on one clock. This single read is what the agent works from, so a log record from another service is not part of it.
• The trigger is per service. In the AI SRE guide, the trigger watches the sources keyed by service and fires when an alert event arrives, counted per service. The agent it wakes reads that service’s correlated timeline and writes a finding back onto it.

Prometheus keeps owning alerting. Tares ingests what fired, correlates it with the evidence and wakes the agent. Nothing is re-thresholded.

Set it up

1. Start Tares and open the console.

   tares up

2. Add an OTLP source keyed by service. There is no required config, and the service label, read from service.name, is the natural key.

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: service, primary: true }

3. Point your exporter, an instrumented app or an OpenTelemetry Collector, at the daemon.

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

   The sender uses a Bearer Authorization header holding an ingest-scoped API key, needed only when Tares has auth on.

4. Add a Prometheus alerts source. Enter the Prometheus URL, run Discover, and optionally filter by severity.

   - name: alerts
     connector: prometheus_alerts
     poll: 30s
     config:
       url: http://localhost:9090
       severities: critical,warning   # blank = all
       labels:
         - { name: severity, field: labels.severity, primary: true }
         - { name: state, field: state }

5. Promote service on the alerts. Discover only proposes the labels that are always present, severity and state. An alert’s own labels, such as namespace and service, exist only while it is firing. Once real alerts have been ingested, open the source’s Fields tab and add service as a label, so alerts and logs share the key.

6. Check the result. In the OTLP source, Recent events shows log records as they arrive and Explore lists services under the service key. In the alerts source, Fields shows the labels your alerts carry.

7. To investigate interactively, start the MCP endpoint in a second shell and connect your client.

   tares mcp

   Then ask, for example, “what happened to api-server in the last 15 minutes?” The client reads the correlated timeline for that service and answers from it.

To put a cheap first-level agent in front of the alert-driven one, see Watch, then escalate.

What you get

• Focused evidence. The agent starts from the logs of the alerting service, next to the alert and the metric that tripped it, instead of a search across everything you ingest.
• One setup for every service. A single OTLP source covers all services, because resource attributes become labels.
• Alerts without a second threshold. Each alert fires once, re-emitting only on a state change, and a resolved event lands on the timeline when it clears.
• Findings that persist. The agent writes its conclusion onto the same service’s timeline, so the next investigation, yours or an agent’s, starts from it.

When something else fits

• If your alerts come through Alertmanager rather than from Prometheus directly, use the alertmanager connector. Keep the same service label so the timeline still joins up.
• If your logs come from containers rather than an OpenTelemetry pipeline, the docker_logs connector can feed the same per-service timeline. Either way the key must match across sources, or the logs will not appear next to the alert.

Related