What should I look for in Docker container logs when the problem is error spikes?

Tares tails a container’s logs into one timeline, so for error spikes you watch ERROR and WARN lines and 5xx request lines against the alert that fired. Tares puts those lines next to the metric and alert for the same service. An agent starts from that correlated timeline instead of from raw lines.

How it works

The docker_logs connector tails a running container’s logs and stores one event per line. It keeps all lines by default. Two optional regex filters shape what lands:

• match keeps only lines that match, for example ERROR|WARN.
• drop skips lines that match, for example HTTP/1.1" to drop access logs.

Give the log source the same service label as your metrics and alerts sources. Tares merges sources that share that label into one timeline per service, so the log lines sit on the same clock as the metric that tripped and the alert that fired.

The telltale lines for an error spike:

• Lines carrying ERROR or WARN, which a match filter keeps.
• Request log lines with 5xx responses. In the AI SRE demo, the error_spike scenario makes the 5xx rate climb, HighErrorRate fires, and the error logs land next to the alert and the metric.
• Lines that start shortly after a deploy. Most real root causes are a change, so a deploy event on the same timeline is worth reading beside the logs.

Counts over a window instead of raw lines. A trigger watches the sources for a service and fires on a condition you choose, aggregated over a window. The demo trigger fires on sum(alert_active) > 0 over 1m, per service. Trigger conditions use an aggregate, a predicate and a window, so a spike is a number crossing a threshold, not something you spot by scrolling. When the trigger fires, it wakes an agent that starts with the correlated timeline already in front of it. The agent writes what broke, since when, the likely cause tied to evidence lines, and a next action. A connected client such as Claude Code asks the same timeline through read for a service, and it sees that finding too.

The broader picture is in Use AI to monitor your systems.

Set it up

1. Install Tares and start it locally. Run taresd on the host, or mount the Docker socket and include the Docker CLI in the image, so Tares can read the container’s logs.

   uv tool install tares          # or: brew install glassflow/tap/tares
   tares up

2. Add the source. Ask a connected agent: “Use tares to find my running containers and ingest the api-server’s logs.” It calls discover_docker, tests the source with a dry run that tails a few lines, then creates it. Or open Sources, Add source, docker_logs in the console and set container to the container or compose service name. Discover lists the running containers to pick from.

3. Set match to keep only the lines that matter for an error spike. Use drop to remove access-log noise if you prefer to keep everything else.

   - name: logs
     connector: docker_logs
     poll: 5s
     config:
       container: tares-demo-api-server   # the container name (docker ps)
       match: 'ERROR|WARN'
       labels:
         - { name: service, const: api-server, primary: true }

4. If the container runs under Compose without a fixed container_name, set container to the service name and add compose_file, an absolute path or one relative to where the daemon runs, so the tail follows restarts.

5. Verify. Open the source in the console. Recent events shows lines on the next poll. If nothing arrives, check that taresd can run docker logs for that container.

6. Put the same service label on your metrics and alerts sources, then add a trigger over them. To investigate yourself, start the MCP endpoint in a second shell:

   tares mcp

The Build an AI SRE guide walks through the full chain with a demo system you can break on purpose, and the Docker logs connector page lists every field.

What you get

• Log lines for the failing container on the same timeline as the metric and the alert, so you read the cause and the effect together.
• A match filter that keeps the error lines and a drop filter that removes access-log noise.
• A trigger that measures the spike over a window, so the agent is woken by a condition and not by someone watching a tail.
• A finding on the service’s timeline that names what broke, since when, and which evidence lines point to the cause. The next investigation, yours or another agent’s, starts from that finding.

When something else fits

• If your logs reach you through an OpenTelemetry pipeline, use the OTLP connector instead of tailing containers. The docker_logs connector needs access to the Docker daemon.
• If an error spike already shows up as a Prometheus alert, the alerts connector gives you the signal directly, and the container logs supply the evidence lines around it.

Related