How does an AI SRE read OpenTelemetry log records while investigating error spikes?

Tares receives your OpenTelemetry logs through the otlp connector and labels them by service, so an agent reads log lines on the same per-service timeline as the alert. When an error spike fires, the agent reads that timeline and sees the error logs next to the alert and the metrics. You connect one otlp source, point your exporter at Tares, and the agent reads the logs through the MCP read surface or when a trigger wakes it.

How it works

The OpenTelemetry connector receives OTLP/HTTP logs, traces and metrics at /v1/logs, /v1/traces and /v1/metrics. One source ingests every service. Resource attributes such as service.name become labels, so everything an OTel SDK or Collector exports correlates by service with no per-service setup.

The service label is the key that ties the signals together. Your metrics source and your alerts source carry the same service label, so Tares builds one timeline per service with logs, metrics and alerts merged on one clock. That is what an agent works from in an AI SRE setup.

An agent reaches that timeline in two ways:

• Pull: you or any MCP client ask about a service. The client calls the read tool with the service name and answers from the correlated read.
• Push: a trigger watching the sources fires when an alert event arrives for a service, and it wakes an agent. The agent reads the correlated timeline, so the error log lines and the alert sit side by side in the same window.

A label reads its value from the event by path:

• resourceAttributes.<key> for resource attributes, for example resourceAttributes.service.name.
• attributes.<key> for the attributes of a log record, span or data point.
• text for the rendered log line.

There is no bare service field, so a label with field: service gets no value from an OTLP event.

Set it up

1. Start Tares locally.

tares up

2. Create an otlp source keyed by service. In the console, go to Sources, then Add source, then otlp. You can also ask a connected agent: “Use tares to create an OTLP source keyed by service.” The agent calls create_source. The source needs no connector-specific config, only the label:

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: resourceAttributes.service.name, primary: true }

3. Point your exporter at Tares. Use OTLP/HTTP, or gRPC if you installed the otlp-grpc extra.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

4. If your instance runs with auth on (tares up --auth), the exporter sends a Bearer Authorization header holding an ingest-scoped API key. Without auth, no header is needed.

5. Verify. Recent events shows logs, spans and metric points as they arrive, and Explore lists your services under the service key.

6. Keep the alert on the same key. Give your alerts source (for example prometheus_alerts or alertmanager) and your metrics source a service label too. The error spike alert then lands on the same timeline as the logs.

7. Connect your agent. Start the MCP endpoint in a second shell and point your MCP client at it.

tares mcp

The endpoint is http://127.0.0.1:8788/mcp. Then ask: “Use tares: what happened to checkout in the last 15 minutes?” Replace checkout with your own service name. For a push path, add a trigger over these sources keyed by service and subscribe an agent to it, as the AI SRE guide shows.

What you get

• Every OpenTelemetry log record from every service arrives through one source, labeled by service.
• During an error spike, the agent reads the error log lines, the alert and the metrics for that service in one correlated timeline, not three separate tools.
• You can investigate on demand from an MCP client, or let a trigger wake an agent when the alert fires.
• Traces and metrics sent over the same endpoint land in the same source and correlate by the same service label.
• The agent writes its conclusion back onto the service’s timeline, so the next investigation starts from the previous finding instead of from scratch.

When something else fits

• If your logs come from containers on a Docker host and you do not run OpenTelemetry, the docker_logs connector tails a container’s logs into the same per-service timeline.
• If a service does not set service.name as a resource attribute, point the label at another resource or record attribute, using the resourceAttributes.<key> or attributes.<key> path form.

Related