How do I line up Grafana Loki logs with Prometheus metrics for one service?

Tares lines up Loki logs and Prometheus metrics when both sources carry the same primary service label. One read of that service then returns the log lines and the metric values merged on one clock. You add a Prometheus source and a Loki source, label both by service, and open the service in Explore or read it from a connected agent.

How it works

Tares builds one timeline per entity key. The key is the source’s primary label, and any sources that share it are merged by time.

• The Prometheus connector polls the metrics you pick and turns each series into a metric event on every poll tick. The numeric value is stored as a number-typed value label.
• The Loki connector polls log lines selected with a LogQL query, every 5 seconds by default. It works with Grafana Cloud too, and it takes the keys url, query, bearer_token, username, password, tenant_id, match, drop and key.
• Both sources use the same label name, service, marked primary, and the same service values. That shared key is what puts a service’s logs and metrics on one timeline.

Tares also keeps the metric value as a number, so a trigger can aggregate it later. This is the same per-service pattern the AI SRE guide uses, where metrics, logs and alerts are all keyed by service.

Set it up

1. Install Tares and start it locally.

uv tool install tares          # or: brew install glassflow/tap/tares
tares up

2. Add the Prometheus source in the console under Sources, Add source, Prometheus. Enter the Prometheus url, use Discover, and pick the metrics for your service by name or by label. Tares generates the config and proposes the labels. Make the primary label service, read from the metric label that names your service. The generated config looks like this:

- name: metrics
  connector: prometheus
  poll: 10s
  config:
    url: http://localhost:9090
    queries:
      - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
        by_name: true
    labels:
      - { name: service, field: metric.service, primary: true }
      - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

3. Add the Loki source from Sources, Add source. Set its url to your Loki instance, and set query to a LogQL selector for the same service’s logs. Add username, password or bearer_token if your Loki needs them. For Grafana Cloud, add tenant_id if it applies to your setup.

4. Give the Loki source a label named service, marked primary, with the same service value your metrics use. Labels take the keys name, field, const, primary and type. If one Loki query covers a single service, a const label with that service name is the simplest choice. The value must match the metric side exactly, for example api-server on both.

5. Check the result. In Explore, pick the service. You see Loki log lines and Prometheus metric events merged by time. Recent events on each source shows its own events, and Fields shows the labels.

6. To read the same timeline from an agent, start the MCP endpoint in a second shell and ask about the service.

tares mcp

What you get

• One timeline per service, with log lines and metric values in time order, so you can see the error burst next to the latency climb that came with it.
• The same read for people and agents. Explore shows it in the console, and a connected MCP client gets it by asking about the service.
• Numeric metric values that triggers can aggregate with max, avg or sum, filtered by event_type.
• A base you can extend. Add alert or deploy sources with the same service label and they land on the same timeline.

When something else fits

• If your logs come from a Docker container rather than Loki, the docker_logs connector tails them directly, and the same shared service label applies.
• If you want a derived metric such as a p99 instead of raw series, the Prometheus connector accepts a computed series with an explicit event_type and field.

Related