How do I line up logs from Docker containers with Prometheus metrics for one service?

Tares lines up Docker logs and Prometheus metrics when both sources carry the same primary service label. One read of that service then returns the log lines and the metric values merged on one clock. You add a docker_logs source and a prometheus source, and give each a primary service label with the same value.

How it works

Every source in Tares can declare labels, and one label is marked primary. The primary label is the entity key. Events from different sources that share the same primary value land on the same timeline for that entity.

• The Docker logs connector tails a running container and stores one event per log line. You set the primary service label with a constant, for example api-server.
• The Prometheus connector turns each series you pick into a metric event on every poll tick. You set the primary service label from the metric’s own service label, which appears as metric.service.
• The metric value is stored as a number-typed value label, so triggers can aggregate it with max, avg or sum.

Because both sources key on service with the same value, a read of api-server returns log lines and metric values together, ordered by time. The same read is what an agent works from in the AI SRE guide.

Set it up

1. Install Tares and start it locally.

   uv tool install tares          # or: brew install glassflow/tap/tares
   tares up

2. Add the logs source under Sources, Add source, docker_logs. Set container to the container name from docker ps, and give it a primary service label. Tares needs access to the Docker daemon, so run taresd on the host or mount the Docker socket. If the container runs under Compose, set compose_file so the tail follows restarts.

   - name: logs
     connector: docker_logs
     poll: 5s
     config:
       container: tares-demo-api-server   # the container name (docker ps)
       labels:
         - { name: service, const: api-server, primary: true }

3. Add the metrics source under Sources, Add source, Prometheus. Enter the url, choose Discover, and pick the metrics by name or by label. There is no PromQL to write. Tares proposes the labels from a sample. Make sure the primary service label reads from the metric’s service label and that its values match the constant you used for the logs, here api-server.

   - name: metrics
     connector: prometheus
     poll: 10s
     config:
       url: http://localhost:9090
       queries:
         - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
           by_name: true
       labels:
         - { name: service, field: metric.service, primary: true }
         - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

4. Verify each source. Open it in the console. Recent events shows lines or series on the next poll. For metrics, Fields shows metric.service and the numeric value. If no logs arrive, check that taresd can run docker logs for that container.

5. Read the service. Open Explore and pick api-server to see logs and metrics merged on one clock. To read it from an agent, start the MCP endpoint in a second shell and ask your client about the service.

   tares mcp

You can also ask a connected agent to do steps 2 and 3. It uses discovery to find the container and the metrics, tests the source with a dry run, then creates it.

What you get

• One timeline per service. Log lines from the container and metric values from Prometheus sit side by side, ordered by time, so you see an error burst next to the metric that moved.
• Metric values you can act on. Because value is number-typed, a trigger can filter to one metric by event_type and aggregate value per service.
• The same view for agents. An MCP client reading the service gets the correlated evidence in one call instead of querying each system separately.
• A pattern that extends. Any further source you key by the same service label, such as alerts from prometheus_alerts, joins the same timeline.

When something else fits

• If your logs come from an OpenTelemetry pipeline rather than a container on the local Docker daemon, use the OTLP connector for logs and keep the same service label on the Prometheus source.
• If you want to compute a derived series such as a p99 or a rate, the Prometheus connector accepts a single PromQL expression with an explicit event_type and field. Use that only when raw metrics are not enough.

Related