How does an AI SRE read logs from Docker containers while investigating a disk filling up?

Tares tails a running container’s logs with the docker_logs connector, one event per line, and labels them by service. When the disk alert fires, the agent reads those log lines on the same per-service timeline as the alert. You connect the container once and every later investigation starts from the same evidence.

How it works

The docker_logs connector tails a running container’s logs and stores one event per line. All lines are kept by default, and you can narrow them with match and drop regex filters.

The key is the service label. When your log source, your metrics and your alerts all carry the same service label, Tares builds one timeline per service. Logs, metrics and alerts sit on one clock. The agent reads that single timeline instead of querying each system.

For a disk filling up, that means:

• The alert event for the disk lands on the service’s timeline, from a prometheus_alerts or alertmanager source.
• The container’s log lines land on the same timeline, so the agent sees what the container wrote around the time the alert fired.
• A trigger on the alert wakes a Tares agent, or your own agent over a webhook. You can also connect an MCP client and ask it to investigate on demand.
• The agent writes its finding back to the timeline, so the next investigation starts from that conclusion.

The AI SRE guide walks through this whole loop end to end. This page covers the log half.

Set it up

Tares needs access to the Docker daemon. Run taresd on the host, or mount the Docker socket and include the Docker CLI in the image.

1. Install the CLI and start Tares.

uv tool install tares      # or: brew install glassflow/tap/tares
tares up

2. Add the log source. In the console, go to Sources, Add source, docker_logs and set container to the container or compose service name. Discover lists the running containers to pick from. You can also ask a connected agent to find your running containers and ingest the logs. The agent calls discover_docker, runs test_source as a dry run that tails a few lines, then calls create_source.

3. Label the source by service. Use the same service label on every source for this system, and mark it primary:

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

4. If the container runs under Compose without a fixed container_name, set container to the service name and add compose_file, an absolute path or one relative to where the daemon runs. The tail then follows restarts.

5. Trim the noise if needed. match keeps only lines matching a regex, for example ERROR|WARN. drop skips lines matching a regex, for example HTTP/1.1" to drop access logs. Both help when a chatty container would bury the lines that explain the disk growth.

6. Give the alert the same service label, so it lands on the same timeline as the logs.

7. Verify. Open the source in the console. Recent events shows lines on the next poll. If nothing arrives, check that taresd can run docker logs for that container, since it needs the same access.

8. Connect an MCP client and ask it about the service.

tares mcp

Then ask: what happened to this service in the last 15 minutes? The client calls read for that service and answers from the correlated timeline, container log lines and alert together.

What you get

• One timeline per service, with the container’s log lines next to the disk alert on the same clock.
• An agent that starts from evidence. It reads the log lines around the alert instead of asking you to paste them.
• Filtering at the source with match and drop, so the timeline holds the lines that matter.
• Log tailing that survives restarts when the container runs under Compose and you set compose_file.
• A finding written back to the timeline, so the next person or agent investigating the same service sees the earlier conclusion.
• Your alerting stays where it is. Tares ingests what fires and handles the correlation and the waking.

When something else fits

• If your logs do not come from Docker containers on the host Tares can reach, use OTLP from your OpenTelemetry pipeline, as the AI SRE guide describes for per-pipeline logs.
• If you only need a quick look at one container once, docker logs directly is simpler. The connector pays off when you want the lines on a timeline with the alert and available to an agent automatically.

Related