How does an AI SRE read OpenTelemetry log records while investigating 5xx errors on an API?

Tares receives your OpenTelemetry log records through one otlp source, labels them by service, and puts them on the same per-service timeline as your alerts. When a 5xx alert lands for your API, the agent reads the log lines from that window together with the alert. You connect the logs once, and every service your exporters send is correlated by its service name.

How it works

• One source for every service. The otlp connector receives OTLP/HTTP logs, traces and metrics at /v1/logs, /v1/traces and /v1/metrics. Resource attributes such as service.name become labels, so everything an OpenTelemetry SDK or Collector exports correlates by service with no per-service setup.
• A service label. The label service reads its value from resourceAttributes.service.name. A label reads from one of three places in the event: resourceAttributes for resource attributes, attributes for the attributes of a log record, span or data point, and text for the rendered log line. There is no bare service field, so a label written as field: service gets no value from an OTLP event.
• One timeline. Alerts, logs and metrics that carry the same service label merge on one clock. Opening the API’s service in Explore shows its log lines next to the alert for the 5xx spike.
• Two ways to read. A trigger can wake an agent when an alert event arrives, and the agent reads the correlated timeline. You can also connect an MCP client and ask about the service yourself, which calls read for that service.

The full walkthrough of the alert, trigger and agent chain is in AI SRE.

Set it up

1. Start Tares locally.

   tares up

2. Create the log source. In the console, go to Sources, then Add source, then otlp. There is no required config. Or ask a connected agent: “Use tares to create an OTLP source keyed by service.” The agent calls create_source. Either way, the source looks like this:

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: resourceAttributes.service.name, primary: true }

3. Point your exporter at Tares. Use the OTLP/HTTP endpoint, or the gRPC endpoint if you installed the otlp-grpc extra.

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

   The sender uses a Bearer Authorization header holding an ingest-scoped API key, needed only when Tares has auth on. Set it through OTEL_EXPORTER_OTLP_HEADERS.

4. Verify the data. Recent events shows logs, spans and metric points as they arrive, and Explore lists your services under the service key. Check that your API’s service name is there.

5. Bring the alert onto the same timeline. Give the source that carries your 5xx alert the same service label, for example prometheus_alerts or alertmanager. Then add a trigger over the log source and the alert source, keyed by service, that fires when an alert event arrives, and subscribe an agent to it.

6. Investigate on demand. Start the MCP endpoint in a second shell and point your client at it.

   tares mcp

   Then ask: “Use tares: what happened to the API service in the last 15 minutes?” The client reads the correlated timeline for that service and answers from the log lines and the alert together.

What you get

• Log lines and the 5xx alert on one clock for the API’s service, so the agent ties the failure to specific log evidence instead of searching a separate log tool.
• One otlp source that covers every service your exporters send, with no per-service setup. A new service shows up under the service key as soon as it exports.
• An agent woken by a trigger with the correlated timeline already in hand, or an MCP client you can question yourself in the same window.
• Extra labels when you need them, read from record attributes with attributes followed by the key name, without changing the service key.

When something else fits

• If your API runs in containers and you only need their stdout, the docker_logs connector tails a container’s logs into Tares without an OpenTelemetry pipeline.
• If you want metrics-based 5xx rates rather than log lines, a Prometheus source gives the agent that signal, and you can run it alongside OTLP under the same service label.

Related