How do I label OpenTelemetry log records so an AI SRE can narrow 5xx errors on an API to one service?

Tares lets you declare a service label on the otlp source, read from the resource attribute service.name and marked primary, so every log record carries a service key. Metrics and alerts keyed by the same service label share that key, so a read on that service returns its 5xx logs, metrics and alerts together. One otlp source covers every service you export.

How it works

The otlp connector receives OpenTelemetry logs, traces and metrics over OTLP/HTTP at /v1/logs, /v1/traces and /v1/metrics. OTLP/gRPC works too with the otlp-grpc extra. One source ingests every service, and you declare labels on it.

A label reads its value from the event by path. There are three places to read from:

• resourceAttributes followed by the key, for resource attributes such as resourceAttributes.service.name.
• attributes followed by the key, for the attributes of a log record, span or data point.
• text, for the rendered log line.

There is no bare service field. A label written as field: service gets no value from an OTLP event, so always use the full path.

Marking a label primary: true makes it the key the event is filed under. When your metrics source and your alerts source use a service label too, Tares builds one timeline per service from all three. Your agent reads that timeline for the API service, so the 5xx log lines arrive next to the 5xx rate and the alerts that fired. For the full picture of how sources, triggers and agents fit together, see Build an AI SRE.

Set it up

1. Create an otlp source. In the console go to Sources, then Add source, then otlp. You can also ask a connected agent: “Use tares to create an OTLP source keyed by service.” The agent calls create_source. Or send the same object with POST /api/sources.

2. Declare the service label from the resource attribute and mark it primary:

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: resourceAttributes.service.name, primary: true }

3. If your exporter does not set service.name, declare the label from a constant instead. A constant fits a source that carries one service, such as the API:

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, const: api-server, primary: true }

4. Name the label service on your metrics and alerts sources too, with the same values. The AI SRE guide keys its Prometheus metrics, container logs and Prometheus alerts sources all by service. The log line, or any record attribute you want to filter on, can be added as further labels using text or the attributes path.

5. Point your exporter at Tares:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

6. When Tares has auth on, the exporter sends a Bearer Authorization header holding an ingest-scoped API key. Without auth you need nothing extra.

7. Verify. Recent events shows logs, spans and metric points as they arrive. Explore lists your services under the service key. Open the API service there to see its logs, metrics and alerts on one clock.

8. Connect an MCP client with tares mcp and ask: “Use tares: what happened to api-server in the last 15 minutes?” The client reads that one service and answers from the correlated evidence.

What you get

• Every log record from the API is filed under its service, so a 5xx investigation starts from one service instead of the whole fleet.
• The same key joins logs with metrics and alerts. A 5xx rate, the error log lines and the alert that fired show up together in one read.
• One otlp source covers all your services. New services appear under the service key as they start exporting, with no per-service setup.
• A trigger keyed by service wakes your agent with that service’s timeline, so the agent works from evidence about the right service.

When something else fits

• If you only need container logs from a single host, the docker_logs connector tails a named container and takes a constant service label. Use OTLP when your services already export OpenTelemetry.
• If your exporter cannot set service.name and one source carries many services, read the label from a record attribute with the attributes path instead of a constant.

Related