What should I look for in Docker container logs when an API returns 5xx errors?

Tares tails a container’s logs with the docker_logs connector, so the error lines behind an API’s 5xx errors sit on one timeline with its metrics and alerts. You keep the request lines and error lines from the API container, and an agent reads that timeline and writes a finding that points at the evidence lines. You open the finding first and go to the cited lines only when you need them.

How it works

The docker_logs connector tails a running container’s logs and stores one event per line. It keeps all lines by default, and you can narrow them with two regex filters, match and drop.

For a 5xx problem, the lines worth keeping are:

• Error and warning lines from the API container. A match of ERROR|WARN keeps only those.
• Request lines, if your API logs them. The drop filter exists to remove access logs, for example with HTTP/1.1", so leave it unset when the request lines are what you want to see.

If your log lines carry fields you want to filter by, label_pattern takes a regex with named groups and turns them into per-line label context.

Give the log source the same service label as your other sources. That label puts the logs, the Prometheus metrics and the alerts on one timeline per service. In the bundled demo, the 5xx rate climbs and the HighErrorRate alert fires. The alert event lands next to the error logs and the metric that tripped it.

A trigger watching those sources can then wake an agent. The agent starts from the correlated timeline, so it does not have to query each system. It writes a finding onto the service’s timeline: what broke, since when, the likely cause tied to evidence lines, and a next action. The finding appears with a headline and the next step, and you can open it for the full note. Use AI to monitor your systems describes this flow in general, and Build an AI SRE walks through it step by step.

Set it up

1. Start Tares and open the console.

   tares up

2. Make sure taresd can reach the Docker daemon. Run it on the host, or mount the Docker socket and include the Docker CLI in the image.

3. Ask a connected agent: “Use tares to find my running containers and ingest the api-server’s logs.” The agent scans your containers, proposes a source, runs a dry-run that tails a few lines, then creates the source. You can also go to Sources, Add source, docker_logs and set container.

4. Set the source so the API container’s error lines are kept and the lines carry a service label. Under Compose without a fixed container_name, set container to the service name and add compose_file.

   - name: logs
     connector: docker_logs
     poll: 5s
     config:
       container: tares-demo-api-server   # the container name (docker ps)
       match: 'ERROR|WARN'
       labels:
         - { name: service, const: api-server, primary: true }

5. Open the source in the console. Recent events shows lines on the next poll. If nothing arrives, check that taresd can run docker logs <container>.

6. Add a Prometheus metrics source and a Prometheus alerts source with the same service label, so the 5xx rate and the alert share a timeline with the logs.

7. To investigate yourself, start the MCP endpoint in a second shell, connect your client, and ask what happened to the service in the last 15 minutes.

   tares mcp

The connector’s own page, Docker logs, lists every field.

What you get

• The API container’s log lines arrive as events, filtered the way you chose, with a service label.
• The error lines sit on one clock with the 5xx rate metric and the alert that fired, so you can see which came first.
• A trigger can wake an agent when the alert arrives. The agent writes a finding with what broke, since when, the likely cause tied to evidence lines, and a next action.
• The finding is on the service’s timeline. The next investigation, yours or another agent’s, starts from it instead of from scratch.
• A connected MCP client answers from the same correlated timeline, so you can ask about the last 15 minutes of the service.

When something else fits

• If the API does not run in Docker, use another log path: OpenTelemetry (OTLP) or Vercel logs feed the same per-service timeline.
• If the 5xx cause is a recent change, add a deploy source as well. The AI monitoring guide calls it the one worth adding first.

Related