How does an AI SRE read logs from Docker containers while investigating a cron job that stopped?

Tares tails a running container’s logs with the docker_logs connector, one event per line, and labels them by service so the agent reads them on the same timeline as the alert. When a cron job stops, the agent sees the alert and the container’s last log lines side by side in one window. You connect the source once, and every investigation after that starts from the same correlated read.

How it works

The Docker logs connector tails a container’s logs and stores one event per line. By default it keeps all lines, and you can narrow them with match and drop regex filters.

The key is the service label. Tares builds one timeline per service from every source that carries the same label. Put the same service value on your Docker logs source and on your alert source, and the log lines and the alert land on one clock. This is the model behind Build an AI SRE.

When a cron job stops, the evidence is usually in its container: the last lines it wrote, an error before it went quiet, or a gap where lines should be. Your alerting keeps deciding that something is wrong. Tares ingests what fired, correlates it with the log lines, and wakes the agent through a trigger. The agent reads the merged timeline, ties its conclusion to specific log lines, and writes a finding back, so the next investigation starts from that conclusion.

The connector needs access to the Docker daemon. Either run taresd on the host, or mount the Docker socket and include the Docker CLI in the image.

Set it up

1. Start Tares and open the console.

   tares up

2. Ask a connected agent to do the wiring: “Use tares to find my running containers and ingest the api-server’s logs.” Swap in the name of your cron container. The agent calls discover_docker to scan running containers, runs test_source as a dry run that tails a few lines, then calls create_source.

3. Or add it by hand. Go to Sources, Add source, docker_logs, set container to the container or compose service name, and use Discover to pick from the running containers. The source looks like this:

   - name: logs
     connector: docker_logs
     poll: 5s
     config:
       container: tares-demo-api-server   # the container name (docker ps)
       labels:
         - { name: service, const: api-server, primary: true }

   Replace the container name with your cron container, and set the service constant to the service the job belongs to. You can also POST /api/sources with the same object.

4. If the container runs under Compose without a fixed container_name, set container to the service name and add compose_file, either an absolute path or one relative to where the daemon runs. The tail then follows restarts.

5. Trim noise if the job is chatty. Use match to keep only lines like ERROR|WARN, or drop to skip lines you do not need. Use label_pattern, a regex with named groups, to turn parts of each line into per-line label context.

6. Give your alert source the same service value. The AI SRE guide shows prometheus_alerts or alertmanager for alerts, with a trigger that wakes an agent when an alert event arrives.

7. To investigate on demand, start the MCP endpoint in a second shell, connect your client, and ask about the service.

   tares mcp

   The client calls read with the service name and answers from the correlated timeline.

8. Verify. Open the source in the console. Recent events shows lines on the next poll. If nothing arrives, check that taresd can run docker logs against that container, because that is the access the connector needs.

What you get

• Every line from the cron container as an event on the service’s timeline, polled every 5 seconds in the example above.
• The alert and the log lines on one clock, so the agent sees what the job printed right before it stopped.
• A woken agent that starts with the correlated timeline already in hand. A connected client reads the same evidence on demand.
• Filtered logs when you want them. match and drop keep the timeline focused on the lines that matter.
• A finding written back onto the timeline, so the next person or agent who investigates this job starts from the previous conclusion.
• Restart-aware tailing when you set compose_file for a Compose container.

When something else fits

• If your logs already flow through an OpenTelemetry pipeline, the AI SRE guide points to OTLP as the log source instead of per-container tailing.
• If the daemon is not reachable from where taresd runs, run taresd on the host or mount the Docker socket first, since the connector depends on that access.

Related