What should I look for in Docker container logs when a disk is filling up?

Tares tails a container’s logs into one event per line, so you can keep the lines that point to a full disk and read them next to your metrics and alerts. Look for write failures such as “No space left on device” and “disk full”, and for the same error repeating from one service. Set a match regex on the docker_logs source to keep only those lines in the window.

How it works

The docker_logs connector tails a running container’s logs and stores one event per line. It keeps all lines by default. Two optional regex filters narrow that down:

• match keeps only lines that match the regex.
• drop skips lines that match the regex, for example access-log noise.

Give the source a service label. Tares merges every source that carries the same service label into one timeline per service, so the container’s log lines sit on the same clock as that service’s metrics and alerts. An agent that reads the timeline, through an MCP client or as a Tares agent woken by a trigger, starts from that correlated view. It does not have to query each system on its own. See Use AI to monitor your systems for how the pieces fit.

The lines worth keeping for a filling disk are the ones where a write fails or a service complains about space:

• No space left on device from the application, database or log writer.
• disk full or similar wording in a service’s own error messages.
• Failed writes or failed log rotation, repeated from the same container.
• The same error line appearing again and again, instead of once.

The exact wording depends on what your containers print, so adjust the regex to match your own logs. If you are unsure which lines your containers emit, tail without a filter first and look at what appears.

Set it up

1. Make sure Tares can reach the Docker daemon. Run taresd on the host, or mount the Docker socket and include the Docker CLI in the image. Start Tares:

tares up

2. Add the source. In the console go to Sources → Add source → docker_logs, set container to the container or compose service name, and use Discover to pick from the running containers. If the container runs under Compose, set compose_file so the tail follows restarts. You can also ask a connected agent: “Use tares to find my running containers and ingest the api-server’s logs.”

3. Add a match filter for the disk lines you want to keep and a service label so the logs join the service’s timeline:

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    match: 'No space left on device|disk full'
    labels:
      - { name: service, const: api-server, primary: true }

4. If you want the surrounding context as well, leave match out and use drop to remove noisy lines instead. That keeps every other line in the window.

5. Verify. Open the source in the console. Recent events shows lines on the next poll. If nothing arrives, check that taresd can run docker logs for that container, since that is the same access the source needs.

6. To investigate with your own MCP client, start the MCP endpoint in a second shell:

tares mcp

Then ask your client about the service, for example what happened to api-server in the last 15 minutes. The walkthrough in Build an AI SRE shows how to connect a client and how a trigger wakes an agent.

What you get

• A log source that holds only the disk-related lines you chose, or every line except the noise you dropped.
• Those lines on one timeline per service, merged on one clock with the service’s metrics and alerts, so you can see what happened around the first failed write.
• An agent that starts from that correlated timeline when you ask it, or when a trigger wakes it.
• A finding written back onto the timeline by a Tares agent, with the likely cause tied to evidence lines and a next action, so the next investigation starts from it.

Details of the source fields are on the Docker logs connector page.

When something else fits

• If the container never logs a disk error, a metric source such as the prometheus connector is a better signal for disk usage than the logs.
• If your logs do not come from a Docker container on a host Tares can reach, the OTLP or Vercel log sources carry logs from other places.

Related