Which logs from Docker containers does an AI SRE search when Alertmanager alert groups fire?

Tares gives an AI SRE the Docker logs of the container you connect, on the same service timeline as your Alertmanager alerts, when both carry the same service label. When an alert group fires, the agent reads that one correlated timeline for the service instead of querying each system separately. The match is made by the service value, so you set it on the alerts and on the logs.

How it works

• Alertmanager POSTs a batch of alerts to the ingest URL of your Tares alertmanager source. Tares fans the batch out into one event per alert. The event type is the alert name, the text is its summary, and the alert labels become Tares labels plus a synthetic status of firing or resolved.
• The alertmanager source keys each alert on the first label it finds in this order: namespace, service, job, instance. An alert that carries namespace lands on the namespace, not on the service. Setting key_label: service on the source makes it key on the service label instead.
• The docker_logs source tails one running container, one event per line. It reads through the Docker daemon, so run taresd on the host. Optional match and drop regex filters keep or skip lines, for example ERROR|WARN to keep, or HTTP/1.1" to drop access logs.
• You give the container’s logs a constant service label, for example api-server. When the service value on the alerts and the service value on the logs are the same, both land on one timeline for that service.
• An agent reads that timeline for the service. In the Build an AI SRE guide, a trigger fires on an alert event, the agent wakes, and it reads the correlated timeline of metrics, logs and alerts for that service. A connected client does the same on demand: asking what happened to api-server in the last 15 minutes makes it call read with the service api-server.

The Alertmanager connector page covers the receiver, and the Docker logs connector page covers the tail.

Set it up

1. Create an Alertmanager source. In the console go to Sources, then Add source, then Alertmanager alerts, name it and create it. Or ask a connected agent to create an Alertmanager source and give you the webhook URL. Set key_label to service so alerts key on the service and not on the namespace:

- name: alerts
  connector: alertmanager
  config:
    key_label: service

2. Add the receiver to your alertmanager.yml with the ingest URL from the source page, then reload Alertmanager:

receivers:
  - name: tares
    webhook_configs:
      - url: http://localhost:8787/ingest/<ingest_key>
        send_resolved: true
route:
  receiver: tares

The sender uses a Bearer Authorization header holding an ingest-scoped API key, needed only when Tares has auth on.

3. Make sure your alert rules put a service label on each alert, with the same value you give the logs in the next step. Send a test alert that carries service: api-server, alongside namespace: prod if you like:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"PostgresDown","severity":"critical","namespace":"prod","service":"api-server"},"annotations":{"summary":"Postgres is down"},"startsAt":"2024-01-01T00:00:00Z"}]}'

4. Create the docker_logs source for the container behind that service. Keep the constant service: api-server label as the primary label:

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

If the container runs under Compose, set compose_file so the tail follows restarts. Add match or drop if you want only certain lines.

5. Check that the service values match. The service on the alert and the service on the logs must be the same string, here api-server, so both land on the same timeline. Open Recent events on each source, then look at the api-server timeline in Explore. The alert and the log lines sit on one clock.

What you get

• Alert groups and container logs on one timeline per service, so the agent starts from the alerting service’s evidence.
• One event per alert, with its labels and status, so the firing and the later resolved both show next to the log lines.
• Line-level control over the log stream with match and drop, so noise such as access logs can stay out of the timeline.
• The same view for an agent woken by a trigger and for a client you ask yourself, because both read the correlated timeline for the service.

When something else fits

• If you want a first level that looks at your sources on a schedule and escalates only when something moves, see Watch, then escalate.
• If your logs come from an OpenTelemetry pipeline instead of container tails, send them with the OTLP connector and give them the same service label.
• If you want Prometheus to own the alerts without Alertmanager’s grouping, use the prometheus_alerts connector instead of alertmanager.

Related