Which logs from Docker containers does an AI SRE search when alerts from Prometheus fire?

Tares gives an AI SRE the logs of the container that carries the same service label as the alert, merged with the alert on one per-service timeline. Instead of searching every container, the agent reads the alerting service and asks for a window such as the last 15 minutes. The alerts come from the prometheus_alerts connector and the logs from the docker_logs connector.

How it works

Two connectors feed one timeline, and a shared label does the narrowing.

• The Prometheus alerts connector polls Prometheus’s own alerts every 30 seconds by default. Each alert becomes an event whose event_type is the alert name and whose text is its summary. It carries the alert’s labels plus a state of firing, pending or resolved.
• Each alert fires once. It emits again only on a state change, and a resolved event arrives when it clears.
• The Docker logs connector tails one named container and stores one event per line. You give the source a service label, for example the constant api-server.
• When the alert source and the log source both carry service, Tares builds one timeline per service. Metrics, logs and alerts sit on one clock.
• The agent reads that timeline for the alerting service. It does not scan the logs of unrelated containers.

You control the window in the question. Asking “what happened to api-server in the last 15 minutes?” makes the connected client read the correlated timeline for that service. The agent sees the first alert event next to the log lines around it.

You can also wake an agent automatically. A trigger on the alert event fires per service, and the agent it wakes reads the correlated timeline and writes a finding with what broke and since when. The Build an AI SRE guide walks through this chain end to end.

Set it up

1. Start Tares and open the console.

   tares up

2. Make sure the Docker daemon is reachable. Run taresd on the host, or mount the Docker socket and include the Docker CLI in the image. The source needs the same access as running docker logs on the container.

3. Add the logs source under Sources, Add source, docker_logs. Set container to the container name from docker ps, or to the Compose service name. If it runs under Compose, also set compose_file so the tail follows restarts. Give it a primary service label.

   - name: logs
     connector: docker_logs
     poll: 5s
     config:
       container: tares-demo-api-server   # the container name (docker ps)
       labels:
         - { name: service, const: api-server, primary: true }

4. Add the alerts source under Sources, Add source, Prometheus alerts. Enter the Prometheus url, run Discover, and optionally filter by severity.

   - name: alerts
     connector: prometheus_alerts
     poll: 30s
     config:
       url: http://localhost:9090
       severities: critical,warning   # blank = all
       labels:
         - { name: severity, field: labels.severity, primary: true }
         - { name: state, field: state }

5. Discover only proposes the severity and state labels, because an alert’s other labels exist only while it fires. Once real alerts have arrived, open the Fields tab and add service (or namespace) so the alert carries the same key as the logs.

6. Optionally narrow the log volume itself. Set match to a regex such as ERROR|WARN to keep only those lines, or drop to skip noise such as access logs.

7. Open Explore, pick the service, and check that alert events and log lines share one timeline. Recent events on each source shows lines on the next poll.

What you get

• An agent that starts from the alerting service, not from every container you run.
• The alert and the log lines around it on one clock, so the first event and its surrounding evidence are read together.
• Alerts that fire once and resolve once, so the timeline is not flooded with repeated copies.
• Optional match and drop filters that keep the stored log lines relevant.
• Findings written back onto the service’s timeline, so the next investigation starts from the previous conclusion.
• The same read for interactive questions: connect an MCP client with tares mcp and ask about the service.

When something else fits

• If you keep alerting in Alertmanager, use the alertmanager connector for the alerts and keep the docker_logs source for logs. The service label does the same narrowing.
• If your logs come through an OpenTelemetry pipeline instead of Docker, use the OTLP connector for them.
• If you want a cheap first look on a schedule before any alert fires, see Watch, then escalate.

Related