How do I label logs from Docker containers so an AI SRE can narrow high latency to one service?

Tares lets you label Docker container logs by declaring a service label on the docker_logs source and marking it primary. The label comes from a constant or a field. Metrics and alerts that carry the same service label share one timeline, so a read on that service returns the log lines next to the latency evidence. See Build an AI SRE for the full walkthrough.

How it works

The docker_logs connector tails a running container and stores one event per log line. It keeps all lines unless you add a match or drop regex.

The labels list on the source decides which entity each line belongs to:

• A label with const gives every line from that container a fixed value, such as service set to api-server.
• A label with field reads the value from a field of the event instead.
• primary: true marks the label that keys the source. Give your metrics and alerts sources the same primary service label and Tares builds one timeline per service from all three.
• label_pattern is a regex with named groups that adds per-line label context.

When latency climbs, that shared key does the narrowing. An agent reads api-server and gets the p99 metric, the HighLatency alert and the container’s request logs merged on one clock. It does not search every container’s output.

The daemon needs access to Docker: run taresd on the host, or mount the Docker socket and include the Docker CLI in the image.

Set it up

1. Find the container name with docker ps. If the container runs under Compose without a fixed container_name, use the Compose service name instead.
2. In the console, go to Sources → Add source → docker_logs and set container. Discover lists the running containers to pick from. For a Compose container, also set compose_file (an absolute path, or relative to where the daemon runs) so the tail follows restarts.
3. Declare the label. This is the source as YAML, and you can also send the same object with POST /api/sources:

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

4. Use the same label name and value on your other sources for that service. The Prometheus connector reads service from the metric labels. The alerts source carries it too. A mismatch in the label name or value puts the logs on a separate timeline.
5. Keep the request lines when latency is the symptom, because they are the evidence. Use drop only for noise, for example HTTP/1.1" to skip access logs you do not need. Use match such as ERROR|WARN to keep only the lines you care about.
6. Verify. Open the source in the console. Recent events shows lines on the next poll. If nothing arrives, check that taresd can run docker logs for that container.
7. Start Tares and the MCP endpoint, then ask your connected agent about the service:

tares up
tares mcp

Ask: “Use tares: what happened to api-server in the last 15 minutes?” The client calls read for the service and answers from the correlated timeline.

What you get

• Every log line from the container is tagged with the service it belongs to, so a latency investigation starts from one service and not the whole fleet.
• Logs, metrics and alerts for that service appear together in Explore and in the read an agent receives.
• A trigger keyed by service wakes an agent with the correlated timeline. You can threshold typed metric fields directly, for example max(p99_ms) > 1000 for latency, or react to fired alerts.
• The agent’s finding is written back onto the same service timeline, so the next investigation starts from the previous conclusion.

When something else fits

• If your logs already flow through an OpenTelemetry pipeline, use the OTLP connector and label the service from a resource attribute.
• If the service does not run in a container the Tares daemon can reach, tailing with docker_logs does not apply, so pick a connector that matches where the logs live.

Related