How do I label Docker container logs so an AI SRE can narrow a stopped cron job to one service?

Tares lets you label Docker container logs with a primary service label, so a read on that service returns the cron job’s logs next to its metrics and alerts. You declare the label on the docker_logs source, from a field or a constant, and mark it primary. Use the same label name and value on your metrics and alert sources. An AI SRE then asks for one service and gets one merged timeline, so a job that went quiet shows up as the point where its log lines stop.

How it works

The docker_logs connector tails a running container’s logs and stores one event per line. Each source can declare labels. A label has a name, and its value comes from a field in the event or from a const. One label is marked primary.

The primary label is the key Tares uses to group events. When your log source, your metrics source and your alerts source all carry the same service label, Tares builds one timeline per service. An agent that reads a service gets the metrics, logs and alerts for it together, on one clock.

For a cron job that stopped, this matters in three ways:

• The job’s log lines carry the service name, so they are not mixed with other containers.
• The alert for the same service lands on the same timeline as the job’s last log line.
• The agent sees where the lines stop instead of having to search each system separately.

Two more settings on the source help you keep the stream clean. match keeps only lines that match a regex, for example ERROR|WARN. drop skips lines that match a regex. For a per-line label context, label_pattern takes a regex with named groups.

Set it up

1. Make sure Tares can reach the Docker daemon. Run taresd on the host, or mount the Docker socket and include the Docker CLI in the image. Tares needs to be able to run docker logs for the container.

2. Start Tares and open the console at http://127.0.0.1:8787.

tares up

3. Go to Sources → Add source → docker_logs. Set container to the name of the container that runs your cron job (see docker ps). Discover lists the running containers to pick from. If the container runs under Compose, also set compose_file so the tail follows restarts.

4. Declare the label. Give it the name your metrics and alerts already use, usually service, set its value from a constant or a field, and mark it primary. The source looks like this:

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

Replace the container name with your cron container and the constant with your service name. A constant fits when one container is one service. When the value should come from the event, declare the label with a field instead of a const.

5. Give your metrics source and your alerts source the same service label with the same value. The Prometheus alerts connector walkthrough in the AI SRE guide keys all three sources by service, and that is what joins them into one timeline.

6. Save and check Recent events on the source. Lines show up on the next poll. If nothing arrives, check that taresd can run docker logs for that container.

7. Connect your agent so it can read the service. Start the MCP endpoint in a second shell and point your client at it.

tares mcp

8. Ask your connected agent a question about the service by name:

Use tares: what happened to api-server in the last 15 minutes?

The client calls read with the service label set to that service and answers from the correlated timeline. Use your own service name in place of api-server. You can also set up the source by asking a connected agent, which calls discover_docker, then test_source, then create_source.

What you get

• One read per service. The cron job’s log lines, its metrics and its alerts come back together, merged on one clock.
• A clear signal for a stopped job. The agent sees the last line the job logged and what the alerts and metrics did around that time.
• Narrowing to one service. Because the primary label is the grouping key, the agent reads only the service you name and not every container on the host.
• Cleaner evidence. With match and drop you keep the lines that matter and skip noise such as access logs.
• A trail for the next investigation. When an agent writes a finding, it lands on the same service timeline, so the next read starts from that conclusion.

For the full path from sources to triggers and agents, see Build an AI SRE. The connector settings are on the Docker logs connector page.

When something else fits

• If your job logs through an OpenTelemetry pipeline instead of container output, the AI SRE guide lists OTLP as the per-pipeline alternative for logs, and the same service label keys it.
• If the container is not running, there is nothing to tail. Fix the container first, then add the source.

Related