What should I look for in logs from Docker containers when the problem is high latency?

Tares tails a Docker container’s logs one event per line, so you can narrow the stream to the lines that matter for latency and read them next to the metric that tripped. Use the match and drop regex filters on the docker_logs source to keep slow-path, warning and error lines and cut routine noise. An agent connected to Tares then starts from the correlated timeline of metrics, alerts and those logs for the service, and writes a finding tied to evidence lines.

How it works

The docker_logs connector tails a running container and stores one event per line. You shape the stream with three options:

• match keeps only lines that match a regex, for example ERROR|WARN.
• drop skips lines that match a regex, for example HTTP/1.1" to drop access logs.
• label_pattern is a regex with named groups that turns parts of a line into per-line label context.

When the problem is high latency, these options decide what an investigator sees. If your access logs carry request duration or status, keep them and use label_pattern to pull those values into labels. If they are only noise, drop them and keep the warnings and errors around the slowdown.

Give the log source the same service label as your metrics and alerts source. Tares merges everything with that label into one timeline per service, so the logs sit on one clock beside the metric that moved and the alert that fired. In the demo, Prometheus rules include HighLatency, and the latency scenario lets you cause a slowdown on purpose and watch the logs line up with the alert.

An agent does not have to query each system separately. A trigger wakes a Tares agent, or your own agent service over a webhook, with the correlated timeline already in front of it. The agent writes what broke, since when, the likely cause tied to evidence lines, and a next action. For numbers, triggers can threshold typed fields over a window, such as max(p99_ms) > 1000 for latency, so the metric side carries the count and the logs carry the explanation.

Set it up

1. Start Tares and open the console at http://127.0.0.1:8787. Make sure taresd runs on the host so it can run docker logs <container>.

tares up

2. Add the source under Sources → Add source → docker_logs, or ask a connected agent: “Use tares to find my running containers and ingest the api-server’s logs.” The agent calls discover_docker, then test_source for a dry run, then create_source.
3. Set container to the container name from docker ps. If the container runs under Compose, set compose_file so the tail follows restarts.
4. Give the source your shared service label as the primary label:

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

5. Add match to keep the lines you care about, such as ERROR|WARN. Add drop if access logs bury the signal. Use label_pattern to capture fields from lines your application writes.
6. Open the source in the console. Recent events shows lines on the next poll.
7. Start the MCP endpoint in a second shell so your client can read the same timeline.

tares mcp

8. Ask your client: “Use tares: what happened to api-server in the last 15 minutes?” It answers from the correlated read, with logs, metrics and alerts for the service together.

To practice, the Build an AI SRE guide stands up a demo system and lets you run the latency scenario end to end. The Docker logs connector page lists every option, and Use AI to monitor your systems shows how logs, metrics, alerts and deploys fit together.

What you get

• A log stream limited to the lines you chose, so a window of logs around a slowdown is short enough to be useful.
• Logs, metrics and alerts for one service on one timeline, so you see which lines appeared when the latency metric or the HighLatency alert moved.
• Per-line labels from label_pattern, so values in your lines become fields instead of text you scan by eye.
• An agent that starts with the correlated timeline already in front of it and writes a finding: what broke, since when, the likely cause tied to evidence lines, and a next action.
• A finding on the timeline that the next investigation, yours or another agent’s, starts from.

When something else fits

• If the evidence you need is a number such as p99, poll it with the Prometheus connector and threshold it in a trigger, and keep the logs for explanation.
• If your services emit OpenTelemetry, the OTLP connector brings logs in through your pipeline instead of tailing a container.
• If the cause may be a recent change, add a deploy source first, since most real root causes are a change.

Related