How do I pair Grafana alert rules with logs from Docker containers for an AI SRE?

Tares pairs Grafana alerts with Docker container logs when both sources carry the same service label, so a firing alert and the log lines from the same window land on one timeline. Grafana reaches Tares through the webhook connector and a Grafana contact point. The docker_logs connector tails the container. Your agent reads the merged timeline for that service, or a trigger wakes it.

How it works

Tares correlates evidence per entity, and the entity is a label you choose. In the AI SRE guide, every source is keyed by the same service label, so metrics, logs and alerts merge on one clock for each service. You do the same here with two sources.

• Grafana alerts arrive through the webhook connector. Tares has no connector of its own for Grafana Alerting today, so you set up a Webhook contact point in Grafana that posts to a Tares ingest URL.
• Docker logs come from the docker_logs connector, which tails a running container and stores one event per line.

A Grafana notification arrives as one event. The full alerts list stays inside the lossless payload. Top-level fields such as status, receiver and title work for labels and for text_template. Because the label you pair on is not one of those top-level fields, give the Grafana source a fixed service label with const, set to the same value as the logs source. Create one Grafana source per service, with its own ingest URL, and point the contact point for that service’s rules at it.

Set it up

1. Install and start Tares if you have not yet. The docs install with uv tool install tares, pipx install tares or brew install glassflow/tap/tares, and you start it with tares up.
2. Create the logs source. Set container to the container name from docker ps, and give it a primary service label. Run taresd on the host, or mount the Docker socket, so Tares can read the container’s logs. Add match or drop regex filters if you want to keep only errors or skip access logs.

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

3. Create the Grafana source as a webhook source with the same service value. Copy its ingest endpoint from the source page.

- name: grafana-alerts
  connector: webhook
  config:
    event_type_field: status
    text_template: 'Grafana {status}: {title}'
    labels:
      - { name: service, const: api-server, primary: true }
      - { name: status, field: status }

4. In Grafana, add a Webhook contact point with the Tares ingest URL and the POST method. When Tares has auth on, set the Authorization header scheme to Bearer and the credentials to an ingest-scoped API key. Without auth on, no header is needed.
5. In Grafana, route the alert rules for that service to the contact point from a notification policy.
6. Check the mapping by posting a sample notification, then open Recent events and Fields on the source page. Replace <ingest_key> with your source’s key.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"receiver": "tares", "status": "firing", "title": "HighErrorRate", "alerts": []}'

7. Open Explore and pick the service. Alert events and log lines appear on one timeline. To have an agent woken when an alert arrives, write a trigger over both sources keyed by service, as the AI SRE guide does, and subscribe an agent to it.

What you get

• One timeline per service, with the Grafana alert and the container log lines from the same window side by side.
• An agent that reads the correlated timeline instead of querying Grafana and Docker separately, whether you connect an MCP client and ask what happened or a trigger wakes an agent.
• Grafana stays in charge of alerting. Tares ingests what fires and correlates it with the evidence, and nothing is re-thresholded.
• The full Grafana payload, including the alerts list, stored lossless, so no detail is lost.
• Log filtering with match and drop, so the timeline carries the lines you care about.

When something else fits

• If your alerts come from Alertmanager, use an alertmanager source. It makes one event per alert and has its own ingest URL, and it can take the same Grafana payload because the default Grafana payload is Alertmanager compatible.
• If your logs come through OpenTelemetry instead of containers on a Docker host, use the OTLP connector and keep the same service label.
• If one Grafana source must cover many services, expect one event per notification with the alerts list inside the payload. Per-service pairing is simplest with one source and contact point per service.

Related