How do I label OpenTelemetry log records so an AI SRE can narrow slow database queries to one service?

Tares lets you declare a label on the otlp source that reads the service name from your log records and mark it primary. Log lines then share a key with that service’s metrics and alerts. A read on the service returns them together, so an AI SRE looks at one service’s slow database queries instead of every log line. The label is one line of config, and no per-service setup is needed.

How it works

One otlp source receives OTLP/HTTP logs, traces and metrics at POST /v1/{logs,traces,metrics} and ingests every service you export. Resource attributes such as service.name become labels, so everything an OpenTelemetry SDK or Collector exports correlates by service.

A label is declared in the source’s labels list. It has a name, a field that says where to read the value, and an optional primary flag. The primary flag marks the key that identifies the entity, here the service. A label reads its value by path from one of three places in the event:

• Resource attributes, by the path resourceAttributes followed by the attribute key. The service name is resourceAttributes.service.name.
• Attributes of a log record, span or data point, by the path attributes followed by the attribute key.
• text, the rendered log line.

There is no bare service field. A label declared with field: service gets no value from an OTLP event, so always use the full path.

If your prometheus metrics, alerts and log lines all carry the same service label, they land on one timeline per service. That is how the AI SRE guide works: an agent reads that timeline and sees the service’s slow-query log lines next to its metrics and alerts.

Set it up

1. Create the source. Ask a connected agent: “Use tares to create an OTLP source keyed by service.” The agent calls create_source. You can also go to Sources → Add source → otlp in the console. There is no required config.

2. Declare the service label and mark it primary. Add this to the source, or send the same object with POST /api/sources:

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: resourceAttributes.service.name, primary: true }

3. Make sure your instrumented services set service.name as a resource attribute. The label reads that value, so every log record from a service carries the same service key as its metrics and alerts. Use the attributes path in the same way if you need another label read from a log record’s own attributes.

4. Point your exporter at Tares. Use the OTLP/HTTP endpoint, or gRPC if you installed the otlp-grpc extra:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

5. If Tares has auth on, the exporter sends an ingest-scoped API key in an Authorization header with the Bearer scheme. Without auth you don’t need it.

6. Verify. Recent events shows logs, spans and metric points as they arrive. Explore lists your services under the service key. Open the service to see its log lines on the same timeline as its metrics and alerts.

The key service must also appear on your metrics and alert sources. Keep it the same everywhere so the read lines up. The OpenTelemetry connector page has the full field reference.

What you get

• Log records from every service you export, keyed by service through one primary label.
• One timeline per service, with logs, metrics and alerts merged on one clock.
• A read on one service that returns its slow-query log lines together with its other evidence. The agent investigates that service and skips the rest.
• The same view for a person in Explore and for an agent over MCP.
• A trigger on that service can wake an agent with the correlated timeline already in hand.

When something else fits

• If the service name is not exported as a resource attribute, set service.name in your SDK or Collector first, because the label reads it from the event.
• For container logs with no OpenTelemetry pipeline, the docker_logs connector tails a container’s output and takes a constant service label. Read the AI SRE guide for how the sources fit together.

Related