How does an AI SRE read OpenTelemetry log records while investigating crash loops?

Tares ingests OpenTelemetry log records through the otlp connector and labels them by service, so an agent investigating a crash loop reads the log lines next to the alert. One otlp source takes every service your exporters send. The agent then works from one correlated timeline per service, whether a trigger wakes it or you ask it through an MCP client.

How it works

• One source for all services. The otlp connector receives OTLP/HTTP logs, traces and metrics at POST /v1/logs, /v1/traces and /v1/metrics. Resource attributes such as service.name become labels, so anything an OpenTelemetry SDK or Collector exports correlates by service with no per-service setup.
• The service label. A label reads its value from a path in the event. The service label uses field: resourceAttributes.service.name. Other resource attributes live under resourceAttributes, attributes of a log record live under attributes, and the rendered log line is text. A label with field: service gets no value from an OTLP event, so always use the full path.
• One timeline per service. Alert sources that carry the same service label, such as prometheus_alerts or alertmanager, land on the same timeline as your log records. The log lines and the alert for a crash looping service sit on one clock.
• Two ways the agent reads it. A trigger fires when an alert event arrives and wakes an agent with the correlated timeline of metrics, logs and alerts for that service. Or you ask a connected MCP client, which calls read for the service and answers from the same timeline. The agent names the likely cause, ties it to specific evidence lines, and writes a finding back to the timeline for the next investigation.

The AI SRE guide builds this loop end to end with a demo system.

Set it up

1. Install Tares and start it locally.

   uv tool install tares          # or: brew install glassflow/tap/tares
   tares up

2. Create the otlp source, keyed by service. Ask a connected agent: “Use tares to create an OTLP source keyed by service.” Or open Sources, Add source, otlp in the console and save this object. There is no required config besides labels.

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: resourceAttributes.service.name, primary: true }

3. Point your exporter at Tares. Set the endpoint for OTLP/HTTP, or use the gRPC address if you installed the otlp-grpc extra.

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

4. If Tares has auth on, the exporter sends a Bearer Authorization header holding an ingest-scoped API key. Without auth on, you need no key.

5. Verify the logs arrive. Recent events shows log records as they come in, and Explore lists your services under the service key. Open the crash looping service and check that its log lines appear.

6. Add the alert side. Create an alert source such as prometheus_alerts or alertmanager and give it the same service label. Then build a trigger over the sources, keyed by service, that fires when an alert event arrives. Choose who it wakes: a Tares agent, your own agent over a webhook, or a Slack channel. The AI SRE guide shows each choice.

7. Connect an MCP client for interactive investigation.

   tares mcp

   Then ask: “Use tares: what happened to my service in the last 15 minutes?” The client reads the correlated timeline, including the log lines the otlp source received.

What you get

• Log records from every instrumented service in one source, labeled by service, with no per-service connector setup.
• A timeline where the alert and the service’s log lines share one clock, so the agent reads the evidence together instead of querying each system separately.
• Investigations that start without anyone polling: the trigger wakes the agent with the timeline already assembled.
• Findings written back to the timeline, so the next investigation, or your own MCP client, starts from the previous conclusion.
• The same OTLP source also takes traces and metrics from the same exporters, and they correlate by the same service label.

See the OpenTelemetry connector page for the full receiver reference.

When something else fits

• If your services only write container logs and have no OpenTelemetry exporter, the docker_logs connector tails a running container’s logs into Tares instead.
• If you want a log source with no label mapping to maintain, check first that your exporters set service.name, because the service label depends on it.
• If you have no alert source yet, you can still investigate by asking an MCP client to read the service’s timeline, then add a trigger later.

Related