How do I label Docker container logs so an AI SRE can narrow 5xx errors on an API to one service?

Tares lets you label Docker container logs by declaring a service label on the docker_logs source and marking it primary: true. The log lines then share a key with that service’s metrics and alerts. A read on that service returns the 5xx rate, the error log lines and the fired alerts together on one timeline, so an AI SRE looks at one service instead of every container.

How it works

The docker_logs connector tails a running container’s logs, one event per line. It needs access to the Docker daemon, so run taresd on the host, or mount the Docker socket and include the Docker CLI in the image.

A label gives each event a key. You declare it under labels with a name and primary: true. In the docker_logs example the value comes from a constant, const: api-server. Every line the source tails carries service set to api-server.

The correlation comes from using the same label name on every source for that service:

• Your metrics source reads service from the metric labels. In the Prometheus connector that is a label with field: metric.service.
• Your alerts source carries the same service key.
• Your Docker logs source sets service to a constant that matches.

Tares merges everything that carries service into one timeline per service, with metrics, logs and alerts on one clock. A read for api-server returns the 5xx rate, the error lines from the API container and any alert such as HighErrorRate. Nothing from other services mixes in. The AI SRE guide builds this whole setup end to end.

If you need a per-line value, the label_pattern option takes a regex with named groups and turns each line into per-line label context.

Set it up

1. Install Tares and start it:

   tares up

2. Find the container name with docker ps. If the container runs under Compose without a fixed container_name, use the service name and also set compose_file to an absolute path, or a path relative to where the daemon runs, so the tail follows restarts.

3. Add the source under Sources → Add source → docker_logs, or ask a connected agent to find your containers and ingest the api-server’s logs. Declare the label with a constant and mark it primary:

   - name: logs
     connector: docker_logs
     poll: 5s
     config:
       container: tares-demo-api-server   # the container name (docker ps)
       labels:
         - { name: service, const: api-server, primary: true }

   You can also send the same object with POST /api/sources.

4. Give your metrics and alerts sources the same service label, so all three share one key. For metrics, read service from the metric labels. The log constant must equal the value those sources produce, here api-server.

5. Verify. Open the source in the console and check that Recent events shows lines on the next poll. If nothing arrives, check that taresd can run docker logs on that container. Then open Explore, pick api-server, and confirm that logs, metrics and alerts appear together.

6. Connect your agent. Start the MCP endpoint in a second shell:

   tares mcp

   Then ask: what happened to api-server in the last 15 minutes? The agent reads the correlated timeline for that one service.

To cut noise before it is stored, set match to keep only lines that match a regex, or drop to skip lines that match one. Both are optional, and the default keeps all lines.

What you get

• Every log line from the API container is tagged with the service it belongs to.
• One read for api-server returns its 5xx rate, its error lines and its alerts on one clock, so the agent ties a spike to specific log evidence.
• Triggers keyed by service wake an agent with the same correlated timeline. The agent’s finding is written back onto that service’s timeline, so the next investigation starts from it.
• Other services’ logs stay out of the read, because they carry a different service value.
• You can check the result yourself in Explore before any agent uses it.

When something else fits

• If your logs already flow through an OpenTelemetry pipeline, the OTLP connector covers logs per service, and you label it with the resourceAttributes.service.name path.
• If your containers run on hosts where taresd cannot reach the Docker daemon, use another source for those logs. See the Docker logs connector for the access it needs.

Related