How do I label OpenTelemetry log records so an AI SRE can narrow a disk filling up to one service?

Tares lets you declare a service label on the otlp source, read from resourceAttributes.service.name and marked primary, so every OpenTelemetry log record carries the service it came from. The log lines then share a key with that service’s metrics and alerts. A read on that service returns the disk symptoms from all three signals together on one timeline.

How it works

One otlp source ingests every service. Tares reads a label’s value from the event by path, and there are three places to read from:

• resourceAttributes followed by the attribute key, for resource attributes such as resourceAttributes.service.name.
• attributes followed by the key, for the attributes of a log record, span or data point.
• text, for the rendered log line.

There is no bare service field. A label declared with field: service gets no value from an OTLP event, so always use the full path.

The label marked primary: true is the key for the entity. When your metrics source (for example Prometheus) and your alerts source carry a service label too, all three line up under the same service. That is how a disk filling up on one service shows its log lines, its metric movement and its fired alert in one place. The AI SRE guide builds this wiring end to end: every source keyed by the same service label gives one timeline per service, with metrics, logs and alerts on one clock.

You can also set a label from a constant with const instead of field. That helps when an exporter does not set service.name and you want a fixed value on the source.

Set it up

1. Start Tares if it is not running yet.

   tares up

2. In the console, go to Sources → Add source → otlp. There is no required config, only labels. You can also ask a connected agent to create an OTLP source keyed by service, or send the same object with POST /api/sources.

3. Declare the service label from the resource attribute and mark it primary:

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: resourceAttributes.service.name, primary: true }

4. Point your exporter, an instrumented app or an OpenTelemetry Collector, at the daemon:

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

   The exporter sends logs to the /v1/logs path of that endpoint. For gRPC you need the otlp-grpc extra, and the receiver listens on port 4317.

5. If your Tares has auth on, the exporter sends a Bearer Authorization header holding an ingest-scoped API key. Without auth on, you need no key.

6. Give the metrics and alerts sources for the same service a service label too, so the keys match. Prometheus metrics can read service from the metric labels, and the alerts sources can do the same with their own label fields.

7. Check the result. Recent events shows log records as they arrive, and Explore lists your services under the service key. Pick the service to see its logs next to its metrics and alerts.

8. Ask a connected client to investigate. Start the MCP endpoint in a second shell:

   tares mcp

   Then ask: “Use tares: what happened to api-server in the last 15 minutes?” The client calls read with the service name and answers from the correlated read, so the log lines that describe the disk filling up arrive with that service’s metrics and alerts.

What you get

• Every log record from the otlp source carries a service label taken from service.name, with no per-service setup.
• One read on a service returns its logs, metrics and alerts together, so an AI SRE can tie the disk symptom to a single service instead of searching across all of them.
• A trigger can watch the sources keyed by service, as the demo trigger does with a per-service condition over a one-minute window. When it fires, the agent it wakes starts from the correlated timeline.
• A conclusion the agent writes back lands on that service’s timeline, so the next investigation starts from it.
• The same label works for traces and metrics that arrive over OTLP, because one source ingests all three.

When something else fits

• If your logs come from a Docker container rather than an OpenTelemetry pipeline, the docker_logs connector tails the container and takes a constant service label.
• If a signal has no service.name and no attribute that names the service, declare a constant label on that source instead of reading a field.
• For the full wiring across metrics, logs, alerts and triggers, follow the AI SRE guide, and see the otlp connector page for every label path.

Related