How does an AI SRE read logs from Docker containers while investigating high latency?

Tares tails a running container’s logs with the docker_logs connector, labels each line by service, and puts those lines on the same per-service timeline as your metrics and alerts. When a latency alert fires, the agent reads that one timeline and sees the log lines from the same window as the alert. You set it up by pointing a source at the container and giving it a primary service label.

How it works

The Docker logs connector tails a running container’s logs and stores one event per line. Every line is kept by default, and you can narrow that with optional regex filters.

The service label is what ties the logs to the rest of your evidence. In the AI SRE setup, the log source, the metrics source and the alerts source all carry the same service label. Tares merges them into one timeline per service on one clock.

The flow during a latency incident looks like this:

• Your alerting rule, such as a HighLatency rule in Prometheus, fires and Tares ingests the alert event.
• A trigger watching the sources, keyed by service, fires on that alert event and wakes an agent.
• The agent reads the correlated timeline for that service. It gets the alert, the latency metrics and the container’s log lines side by side, not three separate queries.
• The agent writes its finding back onto the service’s timeline, so the next investigation starts from that conclusion.

You can also pull instead of waiting for a trigger. A connected MCP client asks about the service, and Tares answers from the same correlated read.

Set it up

1. Install Tares and start it. Tares needs access to the Docker daemon, so run taresd on the host, or mount the Docker socket and include the Docker CLI in the image.

   uv tool install tares     # or: brew install glassflow/tap/tares
   tares up

2. Add the source. In the console go to Sources, Add source, docker_logs and set container to the container or compose service name. Discover lists the running containers to pick from. You can also ask a connected agent to find your running containers and ingest the api-server’s logs. It scans with discover_docker, dry-runs with test_source, then calls create_source.

3. Label the source by service. Give it a primary label named service, and use the same label name on your metrics and alerts sources:

   - name: logs
     connector: docker_logs
     poll: 5s
     config:
       container: tares-demo-api-server   # the container name (docker ps)
       labels:
         - { name: service, const: api-server, primary: true }

4. If the container runs under Compose without a fixed container_name, set container to the service name and add compose_file. Use an absolute path, or one relative to where the daemon runs, so the tail follows restarts.

5. Filter noisy lines if you need to. match keeps only lines that match a regex, for example ERROR|WARN. drop skips lines that match a regex, for example HTTP/1.1" to drop access logs. Be careful with drop when you investigate latency, because request logs often carry the timing evidence. label_pattern takes a regex with named groups and turns them into per-line label context.

6. Verify. Open the source in the console. Recent events shows lines on the next poll. If nothing arrives, check that taresd can run docker logs for that container.

7. Connect your agent. Start the MCP endpoint in a second shell, then point your client at it:

   tares mcp

   Then ask: what happened to api-server in the last 15 minutes? The client reads the correlated timeline for that service, logs included.

8. Add the other two signals with the same service label: a Prometheus metrics source for latency such as p99, and an alerts source for what your rules fire. Then add a trigger over the three sources, keyed by service, so a latency alert wakes the agent.

What you get

• Container log lines for a service sit on the same timeline as its latency metrics and alerts, so the agent reads them in the same window as the alert.
• Each line is one event, labeled by service, so a slow api-server shows its own logs and not those of every container.
• A trigger can wake an agent the moment the alert lands, or you can ask a connected MCP client on demand. Both read the same evidence.
• The agent’s conclusion is written back to the timeline, so the next latency investigation starts from the last finding.
• Filters keep the timeline readable: match for the lines you want, drop for the lines you do not.

When something else fits

• If your logs already flow through an OpenTelemetry pipeline, the OTLP connector is the better fit than tailing individual containers.
• If you only want a narrow slice, such as errors and warnings, keep the docker_logs source but set match so the timeline holds only those lines.

Related