How does an AI SRE read OpenTelemetry log records while investigating a disk filling up?

Tares receives your OpenTelemetry log records over OTLP, labels them by service, and puts them on the same per-service timeline as the alert. An agent reads that one timeline and sees the log lines from the window around a disk alert. You connect one otlp source, point your exporter at Tares, and keep your existing alerting.

How it works

One otlp source ingests every service. It receives OTLP/HTTP logs, traces and metrics at /v1/logs, /v1/traces and /v1/metrics. Resource attributes such as service.name become labels, so whatever an OpenTelemetry SDK or Collector exports correlates by service with no per-service setup.

The label is the join key. If your alert source carries the same service label, the log records and the alert merge on one clock, one timeline per service. The AI SRE guide builds this pattern end to end: logs, metrics and alerts keyed by the same label, then a trigger that wakes an agent.

A label reads its value by path from the event:

• resourceAttributes.<key> for resource attributes, for example resourceAttributes.service.name.
• attributes.<key> for the attributes of a log record, span or data point.
• text for the rendered log line.

There is no bare service field, so a label that uses field: service gets no value from an OTLP event. Use the full path.

For a disk alert, you keep alerting where it is. Prometheus or Alertmanager decides that the disk is filling, and Tares ingests what fired through prometheus_alerts or alertmanager. The agent then reads the alert and the log records for that service together, and you or the agent can narrow to the time window around the alert.

Set it up

1. Install Tares and start it.

uv tool install tares          # or: brew install glassflow/tap/tares
tares up

2. Create the log source. In the console go to Sources, Add source, otlp, or ask a connected agent: “Use tares to create an OTLP source keyed by service.” No connector-specific config is required, only the service label, taken from service.name. The same object works with POST /api/sources.

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: resourceAttributes.service.name, primary: true }

3. Point your exporter at Tares. For OTLP/HTTP use the daemon address. For OTLP/gRPC install the otlp-grpc extra, and the receiver listens on port 4317.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

4. If Tares has auth on, the exporter sends a Bearer Authorization header holding an ingest-scoped API key. Without auth on, you do not need it.

5. Key your alert source by the same service label, so the alert and the log records land on one timeline. The OTLP connector page has the full connector reference.

6. Check the data. Recent events shows logs as they arrive, and Explore lists your services under the service key. Open the service that owns the disk and confirm its log lines sit beside the alert.

7. Connect an MCP client and ask. Start the MCP endpoint in a second shell.

tares mcp

Then ask something like: “Use tares: what happened to api-server in the last 15 minutes?” Use the name of the service whose disk is filling. The client calls read with that service and answers from the correlated timeline.

What you get

• One read for the whole picture. The agent gets the log records and the alert for the service in the same window, instead of querying a log tool and an alert tool separately.
• No per-service wiring. One otlp source covers every service that exports OpenTelemetry data, and the service label groups them.
• Your alerting stays yours. Tares ingests the alert that fired and correlates it with the evidence. Nothing is re-thresholded.
• Pull or push. You can investigate on demand through your MCP client. You can also add a trigger over the same sources so an agent wakes when the alert arrives and writes its conclusion back to the timeline. The next investigation starts from that finding.
• A check you can see. Explore shows the merged timeline, so you can read the same evidence the agent reads.

When something else fits

• If your logs come from a container you run under Docker and not from an OpenTelemetry pipeline, the docker_logs connector tails that container into the same per-service timeline.
• If the agent needs the disk’s own numbers, such as how fast usage is growing, add a prometheus source keyed by the same service label. Log records alone show what the service wrote, not the disk metric.

Related