How can an AI SRE match a jump in Prometheus metrics to the lines in logs from Docker containers?

Tares puts Prometheus metrics and Docker container logs on one timeline per service, so an AI SRE reads both over the same window and sees when the metric moved against when the log lines changed. You give the prometheus source and the docker_logs source the same service label. The agent then calls read for that service and gets metric events and log lines merged on one clock.

How it works

Tares joins signals on a shared key. When the two sources carry the same service label, everything they ingest lands on that service’s timeline.

• Metrics. The Prometheus connector turns each series you pick into a metric event on every poll tick. You choose metrics by name or by a label they carry, and you write no PromQL. The numeric value is stored as a number-typed value label.
• Logs. The Docker logs connector tails a running container and stores one event per line. Optional match and drop regex filters trim the lines, and label_pattern turns named regex groups into per-line label context.
• One read. The agent calls read for the service and receives one correlated window. Metric events and log lines sit in time order, so a rise in the value of a metric can be compared with the lines that appear around the same moment.

The same read is what a woken agent starts from. A trigger over the sources can fire on a condition such as an alert event, and the agent it wakes reads the correlated timeline instead of querying each system separately. The AI SRE guide builds that loop end to end.

Set it up

1. Start Tares and the MCP endpoint so your agent can read.

tares up
tares mcp

2. Add the Docker logs source. Set container to the container name from docker ps, or to the compose service name with compose_file set so the tail follows restarts. Give it a primary service label. Run taresd on the host, or mount the Docker socket, so Tares can run docker logs for the container.

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

3. Add the Prometheus source. Enter the url, use Discover to pick metrics by name or label, and let Tares propose the labels. Make service the primary label, read from the metric labels, so it matches the logs.

- name: metrics
  connector: prometheus
  poll: 10s
  config:
    url: http://localhost:9090
    queries:
      - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
        by_name: true
    labels:
      - { name: service, field: metric.service, primary: true }
      - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

4. Check both sources. Open each one in the console and look at Recent events for lines and series on the next poll.
5. Connect your MCP client and ask: use tares, what happened to api-server in the last 15 minutes? The client calls read for the service and answers from the merged window. In Explore, pick the service to see the same metrics and logs on one clock. Open What the agent gets to see the exact text an agent receives.

You can also ask a connected agent to set up both sources. It uses discover_docker or discover_source, then test_source, then create_source.

What you get

• One window per service that holds the metric events and the container log lines together.
• A clear order of events: the agent sees when the metric value moved and when the log lines around it changed.
• Filters on the log side. Use match such as ERROR|WARN to keep error lines, or drop to skip access logs, so the window stays readable.
• A typed value on every metric event, so a trigger can max, avg or sum it and wake an agent when the metric crosses a limit.
• Findings written back to the service timeline, so the next investigation starts from the previous conclusion.

When something else fits

• If your logs come from an OpenTelemetry pipeline instead of container output, use the OTLP connector and keep the same service label.
• If you want to be woken by alerts Prometheus rules already fire, add the prometheus_alerts source next to these two so the alert lands on the same timeline.

Related