How do I pair Datadog monitors with logs from Docker containers for an AI SRE?

Tares pairs Datadog monitors with Docker container logs by keying both sources with the same service label, so each service gets one timeline. Datadog reaches Tares through the webhook connector and a Datadog webhook setup, and Tares has no connector of its own for Datadog today. The docker_logs connector tails the container, so a firing monitor sits on the same timeline as the log lines from that window.

How it works

• Datadog side. A Datadog webhook posts JSON to a Tares webhook source. You write the payload yourself from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so you decide which fields arrive.
• Docker side. A docker_logs source tails the container and stores one event per log line. It polls every 5 seconds in the example below.
• The shared key. Both sources declare a primary label named service. The webhook source reads it from your payload, and the docker_logs source sets it as a constant. Events with the same service value land on one timeline, merged on one clock.
• Reading it. Open Explore and pick the service to see the alert and the log lines together. A connected agent gets the same view when it calls read with a service. A trigger on the alert source can wake an agent with that correlated timeline. The Build an AI SRE guide walks through triggers and agents end to end.

Set it up

1. Create the webhook source for Datadog with Sources → Add source → webhook, or ask a connected agent. Mark service as the primary label and read it from the payload field of the same name. Keep the payload flat, because event_type_field and text_template read top-level fields only. See Inbound webhook for every option.

- name: datadog-monitors
  connector: webhook
  config:
    labels:
      - { name: service, field: service, primary: true }

2. Copy the ingest endpoint from the source page. Send a test payload that carries the service field and check Recent events.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"service": "api-server"}'

3. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as JSON from Datadog variables, and include a top-level service field set to the same value you use for the container.
4. If Tares has auth on, add custom headers to the Datadog webhook as JSON with an Authorization Bearer header holding an ingest-scoped API key. Without auth, no header is needed.
5. Add the @webhook- handle, followed by your webhook’s name, to the monitor message so the monitor calls it.
6. Add the Docker logs source. Run taresd on the host, or mount the Docker socket and include the Docker CLI in the image, so Tares can read the container’s logs. Set container to the name from docker ps, and add compose_file if the container runs under Compose so the tail follows restarts. The constant service label must match the value your Datadog payload sends. See Docker logs for the optional match and drop filters.

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

7. Trigger a monitor, then open Explore, pick the service and check that the alert and the log lines share one timeline.

What you get

• One timeline per service, with the Datadog alert next to the container’s log lines from the same window.
• An agent that reads the alert and its evidence in a single call instead of querying each system.
• Datadog keeps owning alerting. Tares ingests what the monitor sends and correlates it with the logs.
• The option to put a trigger on the alert source and wake an agent when a monitor fires, as the Build an AI SRE guide shows.

When something else fits

• If your Datadog alerts already flow through Prometheus or Alertmanager, use those connectors instead of a Datadog webhook. They are covered in the AI SRE guide.
• If your containers log through OpenTelemetry rather than the local Docker daemon, send the logs over OTLP and keep the same service label.

Related