How do I label OpenTelemetry log records so an AI SRE can narrow error spikes to one service?

Tares labels OpenTelemetry log records through an otlp source: you declare a service label from the resource attribute service.name and mark it primary. Every log line then carries a service key, the same key your metrics and alerts use. A read on that service returns its logs, metrics and alerts together, so an error spike is narrowed to one service.

How it works

The otlp connector receives OTLP/HTTP logs, traces and metrics. One source ingests every service, so you do not set anything up per service.

A label reads its value from a path in the event. There are three paths:

• resourceAttributes.service.name reads a resource attribute. Other resource attributes follow the same pattern, with the attribute name after resourceAttributes..
• attributes followed by a key reads an attribute of a log record, span or data point.
• text reads the rendered log line.

There is no bare service field. A label declared with field: service gets no value from an OTLP event, so always use the full path.

You can also give a label a constant value with const instead of a field. Use that when everything a source receives belongs to one service.

Marking one label primary makes it the key of the source. When your metrics source and your alerts source use the same primary service label, Tares builds one timeline per service. Logs, metrics and alerts sit on one clock, and that single correlated read is what an agent works from during an error spike. The AI SRE guide shows this wiring end to end, with three sources keyed by the same service label.

Set it up

1. Create the source. In the console go to Sources, then Add source, then otlp. Or ask a connected agent: “Use tares to create an OTLP source keyed by service.” The agent calls create_source. You can also send the same object to POST /api/sources. The OpenTelemetry connector page has the full reference.

2. Declare the label. The otlp source needs no other config, only labels:

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: resourceAttributes.service.name, primary: true }

3. Point your exporter at Tares. Use an instrumented app or an OpenTelemetry Collector:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

For OTLP/gRPC, install the extra with pip install 'tares[otlp-grpc]'. The receiver listens on port 4317.

4. Add authentication if Tares has auth on. The exporter sends a Bearer Authorization header holding an ingest-scoped API key. You need it only when Tares has auth on.

5. Match the key on your other sources. Give your metrics and alerts sources a primary label named service too, so the same service name ties all three together.

6. Verify. Recent events shows logs, spans and metric points as they arrive. Explore lists your services under the service key. Open one service and check that its logs sit next to its metrics and alerts.

7. Ask your agent. Once your client is connected over MCP, ask for one service:

Use tares: what happened to api-server in the last 15 minutes?

The client reads that service and answers from the correlated timeline.

What you get

• Log records from every instrumented service arrive through one source, and each record is labeled with its service name.
• Because the label is primary and shared with your other sources, a read on one service returns its logs, metrics and alerts together. Services you did not ask about stay out of the read.
• A trigger watching sources keyed by service wakes an agent with that one service’s timeline. The agent starts from evidence for the failing service instead of from the whole fleet.
• Findings the agent writes back land on the same service timeline, so the next investigation of that service starts from the earlier conclusion.
• The label comes from your own telemetry, so a new service that exports with a service.name shows up in Explore with no extra setup.

When something else fits

• If your logs are not OpenTelemetry, pick the connector that matches: docker_logs for container logs, or the webhook connector for arbitrary JSON. Keep the same primary service label so they correlate the same way.
• If you need a different grouping than the service, such as a project or environment, declare that label from resourceAttributes or attributes and mark it primary instead. Keep the key identical across the sources you want correlated.

Related