How do I line up Datadog monitors with OpenTelemetry metric series on one timeline?

Tares puts Datadog monitor alerts and OpenTelemetry metric points on one clock by giving both sources the same primary service label. Metrics arrive through an otlp source, and Datadog monitors reach Tares through the webhook connector and a Datadog webhook setup. Open that service in Explore and you see what fired and how the metric got there, side by side.

How it works

Tares correlates events by a shared key. When two sources carry the same service label, everything for that service merges into one timeline, which is the approach in Build an AI SRE.

• Metrics: the OpenTelemetry connector receives OTLP/HTTP logs, traces and metrics. Resource attributes such as service.name become labels, so one otlp source covers every service you export.
• Alerts: Tares has no connector of its own for Datadog today. Datadog reaches Tares through the webhook connector: a Datadog webhook posts each monitor alert to a Tares ingest URL, and the payload is stored lossless.
• The join: the otlp source gets service from service.name. The webhook source reads service from a field in your Datadog payload. When the two values match, both land on the same timeline.

Datadog lets you write the webhook payload yourself from variables such as $ALERT_ID, $EVENT_MSG and $LINK, so you decide which fields the payload carries. Include a field with the service name, spelled exactly like the OTel service.name.

Set it up

1. Start Tares and open the console.

   tares up

2. Add an otlp source keyed by service. In the console go to Sources → Add source → otlp, or ask a connected agent to create it. No other config is needed.

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: service, primary: true }

3. Point your OpenTelemetry exporter or Collector at Tares. Use the first line for OTLP/HTTP, or the second for gRPC, which needs the otlp-grpc extra.

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

4. Add a webhook source for the monitors. Give it a primary service label with the same name as the otlp one. Saving it gives you the ingest URL.

   - name: datadog-monitors
     connector: webhook
     config:
       event_type: monitor_alert
       labels:
         - { name: service, field: service, primary: true }

5. Set up Datadog.

   • Open the Webhooks integration in Datadog and add a webhook whose URL is the Tares ingest URL.
   • Write the payload as flat JSON from Datadog variables, including a service field. Keep it flat, because event_type_field and text_template read top-level fields only.
   • Put @webhook- followed by your webhook’s name in the monitor message, so the monitor calls it.

6. If your payload carries an ISO-8601 timestamp field, name it with event_time_field on the webhook source so events sit at the time Datadog reports.

7. On a secured instance (tares up --auth), both senders need credentials. Add custom headers to the Datadog webhook as JSON with an Authorization Bearer header holding an ingest-scoped API key. Your OTel exporter sends the same kind of key as an Authorization Bearer header. Without auth on, neither is needed.

8. Verify. Recent events on each source shows arrivals, and Fields on the webhook source shows which payload fields come in. Then open Explore and pick the service under the service key.

What you get

• One timeline per service, with Datadog monitor alerts and OTel metric points merged on one clock.
• A clear read of cause and effect: the alert shows what fired, and the metric points before it show how the service got there.
• No per-service setup on the metrics side, because one otlp source ingests every service you export.
• A lossless copy of each Datadog payload, so fields you did not label are still stored.
• The same correlated view for an agent. A trigger can wake it with this timeline, or you can read it from a connected MCP client.

When something else fits

• If your metrics already live in a Prometheus server, the Prometheus connector polls the series you pick, by name or label, with no PromQL to write, and they key on service the same way.
• If a single OTel stream is all you need and nothing alerts from Datadog, the otlp source alone is enough, and you can skip the webhook source.

Related