How do I line up OpenTelemetry log records with Prometheus metrics for one service?

Tares lines up OpenTelemetry logs and Prometheus metrics when both sources carry the same primary service label. One read of that service then returns the log lines and the metric values merged on one clock. You add an otlp source and a prometheus source, key each by service, and open the service in Explore or read it from a connected agent.

How it works

Tares correlates by label, not by query. Each source declares one label as primary, and every event from that source is filed under that value. When two sources use the same label name, such as service, their events for the same service land on one timeline.

• The OpenTelemetry source receives OTLP/HTTP logs, traces and metrics. One source ingests every service. Resource attributes such as service.name become labels, so the service label comes straight from service.name.
• The Prometheus source polls the metrics you pick by name or by label. Each series becomes a metric event on every poll tick. The numeric value is stored as a number-typed value label, so a trigger can aggregate it.
• Because both sources are keyed by service, a read of api-server returns log records and metric points interleaved by time.

This is the same pattern the AI SRE guide uses: every source is keyed by the same service label, and Explore shows metrics, logs and alerts merged on one clock.

Set it up

1. Start Tares and open the console at http://127.0.0.1:8787.

tares up

2. Add the OpenTelemetry source under Sources, Add source, otlp. It needs no connector-specific config, only the primary service label. You can also ask a connected agent to create an OTLP source keyed by service.

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: service, primary: true }

3. Point your exporter, an instrumented app or an OpenTelemetry Collector, at Tares.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

If Tares has auth on, the sender uses a Bearer Authorization header holding an ingest-scoped API key. Without auth, no header is needed.

4. Add the Prometheus source under Sources, Add source, Prometheus. Enter the server URL, run Discover, and tick the metrics you want. Tares proposes the key and labels from a sample, and you never write PromQL. Make sure the primary label is named service, the same name as in the OTLP source.

- name: metrics
  connector: prometheus
  poll: 10s
  config:
    url: http://localhost:9090
    queries:
      - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
        by_name: true
    labels:
      - { name: service, field: metric.service, primary: true }
      - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

5. Check that the label values match. The service name in the OTel resource attribute service.name must equal the value your Prometheus series carries, for example api-server in both. Tares joins on the value, so a mismatch gives you two separate timelines.

6. Verify the result. In Explore, pick your service under the service key. On each source, Recent events shows what is arriving, and Fields on the Prometheus source shows the service and numeric value labels.

7. To read the same timeline from an agent, start the MCP endpoint in a second shell and ask about the service.

tares mcp

What you get

• One timeline per service. Log records from OpenTelemetry and metric values from Prometheus sit on the same clock, so you can see that errors started right after a metric moved.
• No per-service setup on the OTLP side. The one otlp source covers every service your exporters send, and each gets its own timeline through service.name.
• No PromQL on the Prometheus side. You pick metrics, and each event is named by its metric name.
• Metric values you can act on. The number-typed value label can be aggregated with max, avg or sum in a trigger, filtered to one metric by event type.
• One read for agents. A connected MCP client asking about a service gets the correlated logs and metrics together, which is the evidence an AI SRE works from.

When something else fits

• If your logs come from a container rather than an OpenTelemetry pipeline, use the docker_logs source with the same service label. The correlation works the same way.
• If you need a derived metric such as a p99 or a rate, a Prometheus query row can hold a single PromQL expression with an explicit event type and field. Raw metrics need none of that.
• If your OTel service names and Prometheus label values differ, align them at the source, in the exporter or the label mapping, before you rely on the shared timeline.

Related