How do I pair Alertmanager alert groups with logs from Docker containers for an AI SRE?

Tares pairs Alertmanager alerts with Docker container logs when both sources carry the same service label, so they land on one timeline. You create an alertmanager source and a docker_logs source, key both by service, and an agent reads the firing alert next to the log lines from the same window. The AI SRE guide shows the full wiring around it, including triggers and agents.

How it works

Tares builds one timeline per entity key. Sources that carry the same key value are merged on one clock, so an alert and a container’s log lines sit side by side.

• Alertmanager side. The alertmanager connector is a push receiver. Alertmanager has already grouped, deduped and tracked firing to resolved. It POSTs a batch, and Tares fans the alerts array out into one event per alert. The event type is the alert name, the text is its summary, and the alert labels become Tares labels plus a status of firing or resolved. The full alert, including fingerprint, stays in the payload.
• Key derivation. The entity key is taken from a label in this order: namespace, service, job, instance. If your alerts carry namespace, that label wins. Set key_label to service so the alert keys on service instead.
• Docker side. The docker_logs connector tails a running container’s logs, one event per line. A primary label with a constant value, such as service set to api-server, keys every line.
• The match. The alert’s service label value and the log source’s service value must be identical, for example api-server on both.

The Alertmanager connector is a push source, so the sender is Alertmanager. When Tares has auth on, Alertmanager sends a Bearer Authorization header holding an ingest-scoped API key. Without auth you need no header. The Docker connector is polled, so Tares reads from the Docker daemon directly. Run taresd on the host, or mount the Docker socket and include the Docker CLI in the image.

Set it up

1. Start Tares and open the console. Install it with uv tool install tares or brew install glassflow/tap/tares first if you have not.

   tares up

2. Create the Alertmanager source under Sources, Add source, Alertmanager alerts. You can also ask a connected agent to create it and give you the webhook URL. The source page shows a receiver block with the ingest URL filled in. Add it to your alertmanager.yml and reload Alertmanager.

   receivers:
     - name: tares
       webhook_configs:
         - url: http://localhost:8787/ingest/<ingest_key>
           send_resolved: true
   route:
     receiver: tares

3. Make sure your alert rules attach a service label. In the Alertmanager source, set key_label to service so it overrides the namespace default.

   - name: alerts
     connector: alertmanager
     config:
       key_label: service

4. Create the Docker source under Sources, Add source, docker_logs. Set container to the container name from docker ps, or to the Compose service name plus compose_file so the tail follows restarts. Give it a primary service label whose constant matches the alert’s service value.

   - name: logs
     connector: docker_logs
     poll: 5s
     config:
       container: tares-demo-api-server   # the container name (docker ps)
       labels:
         - { name: service, const: api-server, primary: true }

5. Verify the alert path by sending a test batch to the ingest endpoint, then open Recent events. Fields shows the alert labels, so you can confirm which one Tares keys on.

   curl -X POST http://localhost:8787/ingest/<ingest_key> \
     -H 'Content-Type: application/json' \
     -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

6. Verify the log path. Recent events on the Docker source shows lines on the next poll. If nothing arrives, check that taresd can run docker logs for that container.

7. Open Explore, pick your service, and check that alerts and log lines appear on one timeline. To let an agent read it, start the MCP endpoint with tares mcp, connect your client, and ask what happened to the service in the last 15 minutes.

What you get

• One timeline per service. A firing alert shows up with the log lines from the same window, on the same clock, instead of in two tools.
• Resolved state included. With send_resolved: true, a resolved event lands on the timeline when the alert clears.
• Alertmanager stays in charge. Alertmanager keeps owning grouping, silencing and routing. Tares records what it pushes and correlates it with your logs, with no re-thresholding.
• Less noise in the logs. Use the match and drop regex filters on the Docker source to keep only lines like ERROR|WARN, or to skip access logs, so the timeline carries the lines that matter.
• A base for triggers and agents. A trigger over both sources, keyed by service, can wake an agent when an alert event arrives. The agent then reads one correlated timeline and writes its finding back for the next investigation. The AI SRE guide walks through that loop end to end.

When something else fits

• If you want Tares to read alerts that Prometheus rules fire, without Alertmanager’s grouping and silencing, use the prometheus_alerts connector instead. Choose alertmanager when you want grouping, silencing, resolved tracking and real-time delivery.
• If your services do not run as local Docker containers, send logs through the OTLP connector, or through another source you key by the same service label.

Related