How do I pair alerts from Prometheus with logs from Docker containers for an AI SRE?

Tares pairs Prometheus alerts with Docker container logs when you give the prometheus_alerts source and the docker_logs source the same service label. Each service then has one timeline where a firing alert sits next to the log lines from the same moment. An agent reads that timeline, or a trigger wakes it when an alert arrives.

How it works

Tares builds one timeline per service from every source that carries a service label. Two sources feed this pairing:

• The Docker logs connector tails a running container and emits one event per log line. You give it a constant service label, for example api-server.
• The Prometheus alerts connector polls Prometheus’s own alerts endpoint, every 30 seconds in the example. It emits one event per active alert and a resolved event when the alert clears. Each alert carries its Prometheus labels as Tares labels, plus a state of firing, pending or resolved.

Because both sources carry service, Tares merges their events on one clock for that service. Prometheus keeps owning the alerting rules. Tares does not re-threshold anything. It ingests what fired and correlates it with the log evidence.

The Docker logs connector needs access to the Docker daemon. Run taresd on the host, or mount the Docker socket and include the Docker CLI in the image. The service label on an alert exists only while the alert is firing, so you promote it after real alerts have been ingested.

The AI SRE guide builds this pairing end to end with a demo system.

Set it up

1. Add the Docker logs source in Sources → Add source → docker_logs, or ask a connected agent to find your containers. Set container to the container name from docker ps. If the container runs under Compose without a fixed container_name, set container to the service name and add compose_file so the tail follows restarts. Give it the service label:

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

2. Add the alerts source in Sources → Add source → Prometheus alerts. Enter the Prometheus url and click Discover. It proposes the always-present labels, severity and state:

- name: alerts
  connector: prometheus_alerts
  poll: 30s
  config:
    url: http://localhost:9090
    severities: critical,warning   # blank = all
    labels:
      - { name: severity, field: labels.severity, primary: true }
      - { name: state, field: state }

3. Once real alerts have been ingested, open the alerts source and the Fields tab. Promote the per-alert label service so it carries the same value as the constant on the logs source. In the example above, an alert whose service label is api-server lines up with the logs labeled api-server. Use the same spelling and case on both sides.

4. Check both sources. Open each one and look at Recent events. Logs appear on the next poll. If nothing arrives, check that taresd can run docker logs for that container. Alerts appear one per active alert, and resolved events appear as they clear.

5. Open Explore and pick the service. The alert events and the log lines show up merged on one clock.

6. Connect an agent to read the timeline. Start the MCP endpoint, then ask your client about the service:

tares mcp

7. To wake an agent when an alert arrives, add a trigger over both sources, keyed by service. The AI SRE guide uses one that fires when an alert event arrives, per service. Pick who gets woken: a Tares agent, your own agent service over a webhook, or a Slack channel.

What you get

• One timeline per service. A firing alert and the log lines around it appear together, so the agent does not query two systems and line up timestamps itself.
• Alert state changes in the same place. Each alert fires once and re-emits only on a state change. The resolved event lands on the timeline when the alert clears.
• A pull path. A connected MCP client asks what happened to the service and answers from the correlated read.
• A push path. A trigger can wake an agent the moment an alert event arrives. The agent writes its finding back onto the service’s timeline, so the next investigation starts from that conclusion.
• Noise control on the log side. Optional match and drop regex filters keep only the lines you want, for example ERROR|WARN, or drop access logs.

When something else fits

• If your alerts already route through Alertmanager, use the alertmanager connector in place of prometheus_alerts. Keep the same service label.
• If your logs come from an OpenTelemetry pipeline instead of a container on the local Docker daemon, use the OTLP connector for the log side and keep the shared service label.

Related