How do I pair Datadog monitors with OpenTelemetry log records for an AI SRE?

Tares pairs Datadog monitors with OpenTelemetry log records by keying two sources on the same service label. Datadog reaches Tares through the webhook connector and a Datadog webhook setup, and logs arrive through the otlp connector. When a monitor fires, the alert and the log lines from the same window sit on one timeline for that service. Your agent reads that single timeline instead of querying two tools.

How it works

Tares builds one timeline per service. Every source that carries the same service label lands on the same timeline, merged on one clock.

• Logs: the otlp connector receives OTLP/HTTP logs, traces and metrics. One source ingests every service. Resource attributes such as service.name become labels, and the service label is the natural key.
• Monitors: Datadog reaches Tares through the webhook connector and a vendor setup in Datadog. Tares has no connector of its own for Datadog today. A Datadog webhook posts JSON to the ingest URL of a webhook source, and the payload is stored lossless.
• The join: you write the Datadog payload yourself, so you decide that it carries a flat top-level service field. Give it the same value as the service.name your applications export. The webhook source maps that field to a primary service label, and the otlp source does the same. The two streams meet on the service.

Keep the Datadog payload flat. The webhook settings event_type_field and text_template read top-level fields only.

This is the same pattern the AI SRE guide uses. Every source is keyed by one service label, a trigger watches the sources, and the agent reads one correlated timeline.

Set it up

1. Start Tares and open the console at http://127.0.0.1:8787.

   tares up

2. Add the otlp source under Sources, Add source, otlp. It needs no connector-specific config, only the service label.

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: service, primary: true }

3. Point your exporter or OpenTelemetry Collector at Tares. Make sure each service sets service.name.

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

4. Add a second source under Sources, Add source, webhook for the monitors. Declare a service label on the field your Datadog payload carries, and mark it primary. Save, then copy the ingest endpoint from the source page.

   - name: datadog-monitors
     connector: webhook
     config:
       event_type: datadog_alert
       labels:
         - { name: service, field: service, primary: true }

5. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as JSON from Datadog variables, and include a flat service field set to the same name your OpenTelemetry resources use.

6. If Tares runs with auth on, add custom headers to the Datadog webhook as JSON with an Authorization Bearer header holding an ingest-scoped API key. Your OTLP exporters send the same kind of key, for example through OTEL_EXPORTER_OTLP_HEADERS. Without auth on, you need neither.

7. Put @webhook- followed by your webhook name in the Datadog monitor message, so the monitor calls the webhook when it fires.

8. Create a trigger over both sources, keyed by service, that fires when an alert event arrives, and subscribe an agent to it. Datadog keeps owning the alerting. Tares ingests what fired and wakes the agent.

9. Check the result. Recent events on each source shows what arrives, and Fields on the webhook source shows which payload fields come through. In Explore, pick a service and confirm that alert events and log lines appear together.

What you get

• A firing Datadog monitor arrives with the log lines from the same window, on one timeline for that service.
• One otlp source covers every service you instrument. You add no per-service setup on the log side.
• Your agent starts from correlated evidence, so it can tie the alert to specific log lines. You can read the same timeline from a connected MCP client.
• The agent’s conclusion is written back to the service timeline, so the next investigation starts from it.
• The webhook payload is stored lossless, so fields you do not map to labels are still there to read.

When something else fits

• If your alerts come from Prometheus rules or Alertmanager, use the prometheus_alerts or alertmanager connectors instead of a Datadog webhook. They are described in the AI SRE guide.
• If your logs come from a container on the same host, the docker_logs connector tails them without an OpenTelemetry pipeline. Use otlp when you already export OpenTelemetry.

Related