Which logs from Docker containers does an AI SRE search when Datadog monitors fire?

When a Datadog monitor fires, Tares lets your AI SRE read the Docker logs of the alerting service only, on one timeline next to the Datadog event. Datadog reaches Tares through the webhook connector and a vendor setup. The container logs come from the docker_logs connector. Both carry the same service label, so the agent reads one service instead of every container.

How it works

Tares keeps one timeline per service. Two sources feed it, and both carry a service label.

• Datadog monitors reach Tares through the webhook connector and a Datadog webhook setup. Tares has no connector of its own for Datadog today. You write the webhook payload as JSON from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so you decide which fields it carries, including the service name.
• Docker logs come from the docker_logs connector, which tails a running container and stores one event per line.

Because the Datadog event and the container’s log lines share the same service value, they merge on one clock. A trigger on the Datadog source wakes your agent when an alert event arrives. The agent then reads the timeline for the alerting service, so the search is limited to that service’s logs and the alert beside them. When you ask for a window, such as the last 15 minutes, the read covers that window.

To cut volume further, the docker_logs source takes a match regex to keep only some lines, such as ERROR|WARN, and a drop regex to skip noise such as access logs. The AI SRE guide shows the same pattern with a demo system, where alerts, logs and metrics line up per service. Watch, then escalate shows how to put a cheap first-level agent in front of a deeper root-cause agent.

Set it up

1. Start Tares and open the console at http://127.0.0.1:8787.

tares up

2. Add a docker_logs source for the container behind the service. Give it a constant service label that matches the service name your Datadog payload will carry. Set compose_file too if the container runs under Compose, so the tail follows restarts.

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

3. Add a webhook source for Datadog, with a label for the service field your payload carries, marked primary. Keep the payload flat, because event_type_field and text_template read top-level fields only. Copy the ingest endpoint from the source page.

- name: ci-events
  connector: webhook
  config:
    event_type_field: action
    text_template: '{action} by {sender}'
    labels:
      - { name: pipeline, field: pipeline, primary: true }

In your own source, replace the pipeline label with a service label that reads your payload’s service field.

4. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as JSON from Datadog variables. When Tares has auth on, add custom headers as JSON with an Authorization Bearer header holding an ingest-scoped API key. Without auth on, no header is needed.

5. Put the webhook’s handle in the monitor message so the monitor calls it. The handle is @webhook- followed by the name you gave the webhook:

@webhook-<name>

6. Add a trigger over the Datadog source, keyed by service, and subscribe your agent to it. This follows the trigger shape in the guides:

triggers:
  - name: escalate
    sources: [findings]
    key_field: service
    condition: {aggregate: count, predicate: "> 0", window: 5m}
    cooldown: 30m

Point sources at your Datadog source instead of findings. The cooldown keeps one noisy monitor from waking the agent repeatedly for the same service.

7. Fire a test monitor. Open the Datadog source and check Recent events, then open the service in Explore to see the alert and the container’s log lines on one timeline.

What you get

• An agent that starts from the alerting service’s logs and the Datadog event, not a search across every container.
• One correlated timeline per service, with the alert and the log lines on the same clock.
• Log volume you control with match and drop on the docker_logs source.
• A trigger with a cooldown, so the agent wakes once per service instead of once per notification.
• Your own Datadog payload, kept lossless, so extra fields you add are available for labels later.

When something else fits

• If your containers do not run on a host Tares can reach, the docker_logs connector needs access to the Docker daemon, so run taresd on the host or mount the Docker socket. Otherwise send logs through another path such as OTLP.
• If you want a cheap first pass before a deep investigation, use a schedule trigger and handoff as in Watch, then escalate.
• If you already run Alertmanager, its connector gives one event per alert without a vendor webhook setup.

Related