How do I label OpenTelemetry log records so an AI SRE can narrow crash loops to one service?

Tares lets you label OpenTelemetry log records with a primary service label, so an AI SRE can narrow a crash loop to one service. You declare the label on an otlp source from a field or a constant. The log lines then share a key with that service’s metrics and alerts, and a read on that service returns them together.

How it works

One otlp source ingests every service. Resource attributes such as service.name become labels, so what an OpenTelemetry SDK or Collector exports correlates by service with no per-service setup.

A label reads its value from the event by path:

• resourceAttributes. followed by a key reads a resource attribute. The service name is resourceAttributes.service.name.
• attributes. followed by a key reads an attribute of a log record, span or data point.
• text reads the rendered log line.
• A label can also take a fixed value with const instead of a field.

Mark one label primary: true. That label is the key the service’s data is grouped under. When your Prometheus metrics and alerts carry the same service key, the log lines from a crash-looping pod land on the same timeline as the restart alerts and the metrics for that service.

There is no bare service field. A label with field: service gets no value from an OTLP event, so always use the full path.

When your AI SRE asks for one service, for example through the read tool with the service name, it gets metrics, logs and alerts for that service merged on one clock. That is the read the AI SRE guide builds its agents on.

Set it up

1. Create an otlp source. In the console go to Sources, Add source, otlp, or ask a connected agent to create an OTLP source keyed by service. There is no required config beyond labels. You can also POST /api/sources with the same object.

2. Declare the service label from the resource attribute and mark it primary:

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: resourceAttributes.service.name, primary: true }

3. If one source carries a single service, you can declare the label from a constant instead of a field:

labels:
  - { name: service, const: checkout, primary: true }

4. Use the same service key name and the same service values on your metrics and alerts sources, so all three signals share a key. The OpenTelemetry connector page lists the label paths that work for OTLP events.

5. Point your exporter at the Tares daemon:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

6. If Tares has auth on, the exporter sends a Bearer Authorization header holding an ingest-scoped API key. Without auth on, you do not need it.

7. Verify. Recent events shows logs as they arrive, and Explore lists your services under the service key. Open the crash-looping service there and check that its log lines sit next to its alerts and metrics.

What you get

• Log records from a crash-looping service carry a service label, so they are grouped under that service and not mixed with the rest of your fleet.
• The primary label gives logs, metrics and alerts one shared key, so a read on that service returns them together.
• An AI SRE, whether a Tares agent woken by a trigger or your own MCP client, starts from one correlated timeline for the service. It does not have to query logs, metrics and alerts separately and line them up itself.
• One source and one label declaration cover every service your exporters report, because the label reads the service name from each record’s resource attributes.
• The same timeline shows in Explore, so you can look at what the agent sees.

When something else fits

• If the logs come from a container on a host you already run and not from OpenTelemetry, a different connector may fit better. The AI SRE guide shows how each signal is keyed by the same service label.
• If your service name is not in the resource attributes, point the label at the attribute that holds it, using the attributes. path. If a source carries only one service, use a constant.
• If your exporter cannot reach the daemon over OTLP/HTTP, use OTLP/gRPC with the otlp-grpc extra.

Related