How do I line up Grafana Loki logs with OpenTelemetry metric series for one service?

Tares lines up Loki logs and OpenTelemetry metrics when both sources carry the same primary service label. One read of that service then returns the log lines and the metric values merged on one clock. The otlp source takes its service label from the OpenTelemetry resource attribute service.name, and you give the loki source a label with the same name and the same service values.

How it works

Tares correlates by label, not by connector. Every source that carries the same primary service label feeds one timeline for that service, and the timeline holds every signal for it.

• OpenTelemetry metrics. The otlp connector receives OTLP/HTTP logs, traces and metrics at /v1/logs, /v1/traces and /v1/metrics. One source ingests every service. Resource attributes become labels, and service comes from service.name.
• Loki logs. The loki connector polls a Loki instance with a LogQL query, every 5 seconds by default. It also works with Grafana Cloud. You give it a service label so each log line lands on the right service.
• One clock. Both sources write to the same service timeline, so a metric point and a log line from the same moment sit next to each other. This is the same per-service read that the AI SRE guide builds its agent on.

The value is what matters. If the metrics arrive with service.name set to checkout, the Loki label must also resolve to checkout, not to checkout-api or a container name.

Set it up

1. Start Tares and open the console.

   tares up

2. Add the OpenTelemetry source under Sources, Add source, otlp. There is no required config. Make service the primary label.

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: service, primary: true }

3. Point your OpenTelemetry SDK or Collector at Tares. Metrics go to /v1/metrics on the same endpoint as logs and traces.

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

   When Tares has auth on, the exporter sends a Bearer Authorization header holding an ingest-scoped API key. Without auth, no header is needed. See the OpenTelemetry connector page for the details.

4. Add the Loki source under Sources, Add source, loki. Set url to your Loki address and query to the LogQL selector for the service’s logs. If your Loki needs credentials, set bearer_token, or username and password, and set tenant_id for a multi-tenant setup. Then add a label named service, mark it primary, and set its field or const so it produces exactly the value that service.name carries in your metrics.

5. Check that the services match. Open Explore and look under the service key. Your service should appear once, not as two near-identical names, and its timeline should show log lines and metric points together.

6. Read it the way an agent does. Start the MCP endpoint in a second shell and connect your client.

   tares mcp

   Then ask the client about the service, for example “Use tares: what happened to checkout in the last 15 minutes?”. The client calls read with the service name and answers from the correlated timeline.

What you get

• One timeline per service that holds the Loki log lines and the OpenTelemetry metric values on a single clock.
• One read for the whole picture. A person in Explore or an agent over MCP does not query Loki and the metrics store separately and line up timestamps by hand.
• A setup that grows without extra work. The otlp source ingests every service, so a new instrumented service shows up under the service key as soon as it exports. You only add a Loki source, or a Loki query, for the logs.
• The same timeline feeds triggers and agents, so a trigger keyed by service can wake an agent that already sees logs and metrics together.

When something else fits

• If your logs already travel through an OpenTelemetry Collector, send them to the otlp source as well. Logs and metrics then share one source and one service label, and you skip the Loki source.
• If you only need the metrics, or only the logs, one source is enough. Add the second when you want them on the same timeline.

Related