What should I look for in OpenTelemetry log records when the problem is high latency?

Tares puts your OpenTelemetry log records on one timeline per service, next to metrics, alerts and deploys, so you check the log line, the record attributes and the service label together. An agent reads that correlated timeline and writes a finding with what broke, since when, the likely cause tied to evidence lines, and a next action. You read the finding instead of paging through raw lines.

How it works

The otlp connector receives OTLP/HTTP logs, traces and metrics. One source ingests every service, and resource attributes such as service.name become labels, so everything your exporter sends correlates by service with no per-service setup. For a latency problem, three parts of a log event matter:

• The log line. A label can read the rendered line with field: text. This is where you look for lines about slow or timed-out work.
• Record attributes. Labels read the attributes of a log record, span or data point with a path that starts with attributes.. Use this for durations or route names that your app attaches to its records.
• Resource attributes. These use a path that starts with resourceAttributes.. The service label is field: resourceAttributes.service.name. A label with field: service gets no value from an OTLP event, so always use the full path.

The log lines are only one part of the picture. Tares merges them with the other sources you label with the same service: metrics from Prometheus, alerts, and deploys. The demo in the AI SRE guide ships a HighLatency alerting rule and a latency scenario you can trigger on purpose. For your own systems, a trigger can threshold a typed field directly, for example max(p99_ms) > 1000 for latency.

When a trigger fires, a Tares agent, or your own agent service over a webhook, starts with the correlated timeline already in front of it. It does not spend its first minutes querying each system. It then writes its finding back onto the service’s timeline, and the next investigation starts from that finding.

Set it up

1. Install Tares and start it locally.

uv tool install tares      # or: brew install glassflow/tap/tares
tares up

2. In the console, go to Sources → Add source → otlp, or ask a connected agent: “Use tares to create an OTLP source keyed by service.” There is no required config. Add the service label so the logs land on the same timeline as your metrics and alerts.

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: resourceAttributes.service.name, primary: true }

3. Point your exporter at the daemon. OTLP/HTTP goes to /v1/logs, /v1/traces and /v1/metrics. For OTLP/gRPC, install the otlp-grpc extra, and the receiver listens on port 4317.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

The sender uses a Bearer Authorization header holding an ingest-scoped API key, needed only when Tares has auth on.

4. Give your other sources for the same service the same service label: metrics, alerts and a deploy webhook. The AI monitoring guide lists the connector for each signal. Deploys are the source worth adding first, because most real root causes are a change.

5. Start the MCP endpoint in a second shell so your own client can read the same timeline.

tares mcp

6. Check Recent events to see log records arrive, and Explore to see your services under the service key. Then ask your connected client what happened to the service in the last 15 minutes. It answers from the correlated read.

What you get

• One timeline per service that holds your OpenTelemetry log lines beside the latency metric, the alert that fired and any deploy.
• Log labels that read the line (text), the record attributes (attributes. paths) and the resource attributes (resourceAttributes. paths), so you can key and filter on the fields that describe slow work.
• A trigger that wakes an agent when your latency condition holds, with the correlated timeline already in front of it.
• A finding on the timeline that names what broke, since when, the likely cause tied to evidence lines, and a next action. You open the finding first and go to the raw lines only to check the evidence it points at.
• A connected MCP client that reads the evidence and the earlier finding, so a follow-up investigation starts from the previous conclusion.

When something else fits

• If your logs come from containers on one host, the Docker logs connector tails them without an OpenTelemetry pipeline. If they come from Vercel, use the Vercel connector.
• If you only need a raw latency number, a typed metric from Prometheus with a trigger threshold is the more direct signal. Use the logs to explain the number.

Related