What should I look for in OpenTelemetry log records when the problem is a disk filling up?

Tares collects your OpenTelemetry log records by service, so you can spot disk-full lines such as no space left on device on the timeline of the service that wrote them. Look for write failures, out-of-space messages and the same error repeating from one service. An agent connected to Tares reads that service’s timeline and writes a finding that ties the cause to specific evidence lines.

How it works

Tares receives OTLP/HTTP logs, traces and metrics at POST /v1/{logs,traces,metrics}. One otlp source takes every service, and the resource attribute service.name becomes the service label. Your log records then sit on the same per-service timeline as your metrics and alerts.

When a disk is filling up, these are the lines worth looking for in the log text:

• Out-of-space messages. Phrases such as no space left on device, disk full or out of disk space are the clearest sign.
• Write failures. Errors from a service that fails to write a file, a log, a temp file or a database page.
• Repetition. The same message arriving again and again from one service, which matters more than a single stray line.
• Timing. Whether the first such line appears after a deploy or a change, which you can see because deploys land on the same timeline.

The rendered log line is available to labels as text. Record attributes are available as attributes followed by the key name, and resource attributes as resourceAttributes followed by the key name. There is no bare service field, so a label that uses field: service gets no value from an OTLP event.

A trigger can watch the source for a service and fire on a condition you choose. Its condition counts events over a window, with an aggregate, a predicate and a window length. The agent it wakes starts with the correlated timeline already in front of it, then writes what broke, since when, the likely cause tied to evidence lines and a next action.

Set it up

1. Create an otlp source keyed by service. In the console, go to Sources, then Add source, then otlp, and save this config. You can also ask a connected agent: use tares to create an OTLP source keyed by service.

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: resourceAttributes.service.name, primary: true }

2. Point your exporter at the daemon. OTLP/HTTP goes to port 8787. OTLP/gRPC needs the otlp-grpc extra and listens on port 4317.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

3. If Tares has auth on, the exporter sends a Bearer Authorization header holding an ingest-scoped API key. Without auth, you need no key.

4. Start Tares and, in a second shell, the MCP endpoint your agent reads from.

tares up
tares mcp

5. Check that logs arrive. Recent events shows log records as they land, and Explore lists your services under the service key. Open the service you suspect and scan its timeline for the lines above.

6. Ask your connected agent about the service and a time range, for example: what happened to api-server in the last 15 minutes? It calls read for that service and answers from the correlated timeline, including any earlier finding. The Build an AI SRE guide walks through connecting a client and wiring a trigger to an agent end to end.

For the full list of fields the otlp source accepts, see the OpenTelemetry connector page.

What you get

• Log records from every service on one source, labeled by service, with no per-service setup.
• Disk-full lines on the same timeline as the service’s metrics, alerts and deploys, so you see what changed around the first out-of-space message.
• A trigger that can count events over a window and wake an agent when your condition holds.
• An agent that starts from the correlated timeline and writes a finding with the likely cause tied to evidence lines and a next action, so the next investigation starts from that finding.

When something else fits

• If the signal is a disk metric rather than a log line, a Prometheus source for that metric fits better, and Use AI to monitor your systems shows how to put it on the same timeline.
• If your logs come from a container rather than an OpenTelemetry pipeline, the Docker logs connector reads the container’s output directly.

Related