How does an AI SRE read OpenTelemetry log records while investigating high latency?

Tares receives your OpenTelemetry log records over OTLP, labels them by service, and puts them on the same per-service timeline as the latency alert. The agent reads that one timeline, so the log lines around the alert sit next to it. You connect one otlp source keyed by service and point your exporter at Tares.

How it works

• One source for every service. The otlp connector receives OTLP/HTTP logs, traces and metrics at /v1/logs, /v1/traces and /v1/metrics. Resource attributes such as service.name become labels, so everything an OTel SDK or Collector exports correlates by service with no per-service setup.
• The service label. A label reads its value by path from the event. The service label uses resourceAttributes.service.name. Attributes of a log record, span or data point sit under attributes, and the rendered log line is text. There is no bare service field, so a label with field: service gets no value from an OTLP event.
• One timeline per service. When your metrics, logs and alerts all carry the same service label, Tares merges them on one clock. Open Explore, pick the service, and you see the latency alert, the metric that tripped it and the log lines from the same window.
• Two ways the agent reads it. A trigger watching the sources wakes an agent when an alert event arrives, and the agent reads the correlated timeline. Or you connect an MCP client and ask about the service yourself. Both read the same evidence. See Build an AI SRE for the full walkthrough, including the demo’s HighLatency alert.

Set it up

1. Start Tares and open the console at http://127.0.0.1:8787.

   tares up

2. Create the OTLP source. In the console go to Sources, Add source, otlp, or ask a connected agent: “Use tares to create an OTLP source keyed by service.” There is no required config. Save this source:

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: resourceAttributes.service.name, primary: true }

3. Point your exporter at Tares. OTLP/HTTP goes to port 8787. For OTLP/gRPC, install the otlp-grpc extra and use port 4317.

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

   The sender uses a Bearer Authorization header holding an ingest-scoped API key, needed only when Tares has auth on.

4. Check that data arrives. Recent events shows logs, spans and metric points as they come in, and Explore lists your services under the service key.

5. Key your alert source by the same service label. Use the prometheus_alerts or alertmanager connector, so the latency alert lands on the same timeline as the logs. Keep your alerting where it is. Tares ingests what fires.

6. Choose who investigates. A trigger over your sources, keyed by service, wakes a Tares agent when the alert arrives. You can also threshold typed fields directly, such as max(p99_ms) > 1000 for latency. To investigate on demand, start the MCP endpoint and point your client at it.

   tares mcp

   Then ask, for example: “Use tares: what happened to checkout in the last 15 minutes?” Replace checkout with your service name. The client answers from the correlated read.

The OpenTelemetry connector page has the full configuration and verification details.

What you get

• Log lines beside the alert. When a latency alert fires, the agent reads the log records from the same window on one timeline, instead of querying a log tool on its own.
• Service-level correlation without extra setup. One otlp source covers every service that exports to it, so a new service shows up on the timeline as soon as its resource attributes carry a service name.
• Evidence tied to a conclusion. A triggered agent takes the first look, ties the failure to specific evidence lines and suggests a next action. It writes the finding back onto the service’s timeline, so the next investigation starts from that conclusion.
• The same read for you. Your MCP client sees the same timeline and the same findings, so you can check the agent’s reasoning or continue the investigation yourself.

When something else fits

• If your logs come from a container rather than an OpenTelemetry pipeline, the docker_logs connector tails a running container’s logs into the same per-service timeline.
• If you only need to know that latency crossed a threshold, a trigger on the metrics or alert source is enough, and you can add the log source when you want line-level evidence.
• If your exporter cannot reach the Tares daemon directly, send through an OpenTelemetry Collector that can.

Related