How do I label OpenTelemetry log records so an AI SRE can narrow high latency to one service?

Tares labels OpenTelemetry log records through a primary service label on the otlp source, so an AI SRE can read one service’s logs, metrics and alerts together. You declare the label from a field in the event, mark it primary, and give your metrics and alerts sources the same label key. A read on that service then returns everything for it on one timeline, which is where a latency problem shows up.

How it works

One otlp source ingests every service. Each OTLP log record, span and metric point becomes an event, and a label pulls its value out of that event by path.

• resourceAttributes followed by the key reads a resource attribute. The service name is resourceAttributes.service.name.
• attributes followed by the key reads an attribute of a log record, span or data point.
• text reads the rendered log line.
• A label can also take a fixed value with const instead of a field.

A label marked primary is the key that groups events. When your metrics source and your alerts source use the same label name, here service, their events share a key with the log lines. Tares then builds one timeline per service, with metrics, logs and alerts merged on one clock.

There is no bare service field in an OTLP event. A label written as field: service gets no value, so always use the full path resourceAttributes.service.name.

The AI SRE guide builds on this. Its sources are all keyed by the same service label, so a trigger or a connected agent reads one correlated view of the service instead of querying each system separately.

Set it up

1. Create the otlp source. In the console go to Sources, Add source, otlp. There is no required config except the labels. You can also ask a connected agent: use tares to create an OTLP source keyed by service. Save this definition:

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: resourceAttributes.service.name, primary: true }

2. Add more labels if they help the investigation. Read a resource attribute with resourceAttributes and its key, or a record attribute with attributes and its key. Use const for a value that never changes. Keep exactly one label primary.

3. Point your exporter at Tares. Use the OTLP/HTTP endpoint, or the gRPC endpoint if you installed the otlp-grpc extra:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

4. If your Tares instance has auth on, the exporter sends a Bearer Authorization header holding an ingest-scoped API key. Without auth, you need no key.

5. Give your metrics and alerts sources a label with the same name, service, marked primary. Their values must match the service names your logs carry, so a latency alert and the log lines for that service land under one key.

6. Verify. Recent events shows logs, spans and metric points as they arrive, and Explore lists your services under the service key. Pick the service that alerts on high latency and check that its logs, metrics and alerts sit together.

7. Start the MCP endpoint with tares mcp and ask your connected client, for example: use tares, what happened to checkout in the last 15 minutes? The client reads that one service and answers from the correlated timeline.

What you get

• A single read per service. When latency climbs, the AI SRE sees the slow-request log lines next to the latency metric and the alert that fired, instead of searching each system.
• Narrowing to one service. Because the primary label carries the service name, an agent asks about checkout and gets only that service’s evidence.
• Correlation with no per-service setup. One otlp source covers every service your exporters send, and each new service shows up under the service key as its logs arrive.
• Triggers that wake an agent with the same timeline. A trigger keyed by service can fire on an alert and hand the agent the merged evidence, and the agent can write its finding back onto that service’s timeline for the next investigation.
• A working check in the console. Explore shows the service list, so you can confirm the label came through before you rely on it.

When something else fits

• If a service’s logs are plain container output rather than OpenTelemetry, the docker_logs connector tails a container and takes a constant service label, which also correlates with the other sources.
• If a record carries the service name only in an attribute and not in the resource, point the label at attributes plus that key instead of resourceAttributes.service.name.
• If you only need to see raw events as they arrive, Recent events on the otlp source is enough, and no labels beyond the primary one are needed.

Related