How can an AI SRE match a jump in Prometheus metrics to the lines in OpenTelemetry log records?

Tares puts Prometheus metrics and OpenTelemetry log records on one timeline per service, so an AI SRE reads a single window and sees when the metric moved against when the log lines changed. Key both sources by the same service label. The agent then calls read for that service through MCP and lines up the numeric value events with the log lines on one clock.

How it works

Both sources carry a service label, and that shared key is what joins them.

• Metrics. The Prometheus connector polls the metrics you pick, by name or by a label they carry, with no PromQL to write. Each series becomes one metric event per poll tick. The numeric value is stored as a number-typed value label.
• Logs. The OTLP connector receives OpenTelemetry logs, traces and metrics at /v1/logs, /v1/traces and /v1/metrics. One source ingests every service, and resource attributes such as service.name become labels. The service label comes from service.name.
• One window. Because metric events and log records share the service key, they merge into one timeline for that service. A read covers one window, so the metric points and the log lines sit next to each other in time.
• The agent. A connected agent calls read with the service set to the one you care about, for example api-server, and answers from the correlated result. The agent does the matching by reading the timeline: it sees when value climbed and which log lines appeared or changed around the same moment.

The AI SRE guide builds this whole loop end to end: sources keyed by service, a trigger, an agent, and a finding written back to the timeline.

Set it up

1. Install and start Tares, then start the MCP endpoint in a second shell so your agent can read.

tares up
tares mcp

2. Create the OTLP source. In the console go to Sources → Add source → otlp, or ask a connected agent to create an OTLP source keyed by service. There is no required config beyond the label.

- name: otel
  connector: otlp
  config:
    labels:
      - { name: service, field: service, primary: true }

3. Point your OpenTelemetry exporter at the daemon.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
# or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

The sender uses a Bearer Authorization header holding an ingest-scoped API key, needed only when Tares has auth on.

4. Create the Prometheus source. In the console go to Sources → Add source → Prometheus, enter the url, choose Discover, and pick the metrics to ingest. The console and agent generate the config for you. Make sure the primary label is service, read from the metric labels.

- name: metrics
  connector: prometheus
  poll: 10s
  config:
    url: http://localhost:9090
    queries:
      - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
        by_name: true
    labels:
      - { name: service, field: metric.service, primary: true }
      - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

5. Check both sources. Recent events shows logs and metric points as they arrive. Explore lists services under the service key, and picking one shows its metrics and logs merged on one clock. The What the agent gets view shows the exact text an agent receives.

6. Connect your MCP client and ask: “Use tares: what happened to api-server in the last 15 minutes?” The client calls read for that service and answers from the merged window.

What you get

• One window per service with metric events and log records on the same clock, so the agent can say the metric rose at one time and the log lines changed at another.
• A numeric value on every metric event, so a trigger can aggregate it with max, avg or sum and wake an agent when the jump happens.
• No per-service setup on the log side. Every service that exports OTLP lands in the one source and correlates by service.
• Findings the agent writes back onto the same timeline, so the next investigation starts from the previous conclusion.
• The same view for you in Explore, so you can check the agent’s reading against the raw events.

When something else fits

• If your logs come from containers and not an OpenTelemetry pipeline, the Docker logs connector keys the same service label, and the read works the same way.
• If you want Prometheus to keep owning the threshold and only need to know what fired, ingest the alerts with the Prometheus alerts connector and let the trigger wake the agent from those.
• If the cause is a change rather than a load pattern, add a webhook source for deploys keyed by service, so the deploy lands on the same timeline.

Related