How do I line up Datadog monitors with Prometheus metrics on one timeline?

Tares puts Datadog monitor alerts and Prometheus metrics on one timeline when both sources carry the same primary service label. Datadog reaches Tares through the webhook connector and a Datadog webhook setup, and Prometheus is polled by its own connector. You then read the alert and the metric that led to it on one clock for each service.

How it works

Tares builds one timeline per entity key. Every source that carries the same primary label, here service, lands on the same timeline for that service.

• Prometheus metrics come in through the Prometheus connector. You pick metrics by name or label, and Tares polls them on an interval. Each series becomes a metric event on every poll tick. The value is stored as a number-typed value label, so you never write PromQL.
• Datadog monitors come in through the webhook connector. Tares has no connector of its own for Datadog today. A Datadog webhook posts JSON to a Tares ingest URL, and the webhook source maps your payload fields into labels.
• The join is the label. The Prometheus source reads service from the metric labels, and the webhook source reads service from your payload. Both mark it primary, so events from both sit side by side, ordered by time.

The Build an AI SRE guide shows the same pattern end to end: metrics, logs and alerts keyed by service merge into one view per service.

Set it up

1. Add the Prometheus source. In Sources → Add source → Prometheus, enter the server URL and pick your metrics. The console generates a config like this:

   - name: metrics
     connector: prometheus
     poll: 10s
     config:
       url: http://localhost:9090
       queries:
         - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
           by_name: true
       labels:
         - { name: service, field: metric.service, primary: true }
         - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

2. Add a webhook source for Datadog. In Sources → Add source → webhook, declare a service label as primary. Save, then copy the ingest endpoint from the source page.

   - name: datadog-monitors
     connector: webhook
     config:
       event_type: monitor_alert
       text_template: '{message}'
       labels:
         - { name: service, field: service, primary: true }

3. Create the webhook in Datadog. Open the Webhooks integration and add a webhook whose URL is the Tares ingest URL.

4. Write the payload. In the webhook Payload field, write JSON built from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK. Include a top-level service field whose value matches the service value on your Prometheus series. Keep the payload flat, because event_type_field and text_template read top-level fields only.

5. Add the credential if Tares has auth on. Add custom headers as JSON with an Authorization Bearer header that holds an ingest-scoped API key.

6. Call the webhook from the monitor. Put @webhook- followed by your webhook name in the monitor message, so the monitor calls it.

7. Test the mapping. Post a sample payload to the ingest URL and check Recent events and Fields on the source:

   curl -X POST http://localhost:8787/ingest/<ingest_key> \
     -H 'Content-Type: application/json' \
     -d '{"service": "api-server", "message": "High error rate on api-server"}'

8. Read the timeline. Open Explore and pick the service. Prometheus metric events and Datadog alert events appear on the same clock.

What you get

• One timeline per service that shows what fired (the Datadog monitor alert) next to how the metric got there (the Prometheus series).
• Metric values stored as a number-typed value label, so a trigger can aggregate them with max, avg or sum while the alert sits on the same timeline.
• Lossless Datadog payloads. The webhook connector stores the whole JSON, and your labels pull out the fields you correlate on.
• The same evidence for an agent. A connected agent or a trigger-woken agent reads the correlated timeline, as described in Build an AI SRE.
• Checks you can run. Recent events shows one event per series per poll for Prometheus, and Fields shows which payload fields arrive for the webhook.

When something else fits

• Your alerts live in Prometheus. If Prometheus rules fire the alerts, use the prometheus_alerts or alertmanager connector with the Prometheus metrics. Those alerts reach Tares without a vendor webhook setup.
• You only need the metric. If you don’t need Datadog alerts on the timeline, the Prometheus connector alone gives you metric events per service.
• Your service names differ between systems. The join needs the same service value on both sides. If Datadog and Prometheus name a service differently, set the value you send in the Datadog payload to match the Prometheus label, or the events stay on separate timelines.
• You need other payload shapes. The webhook connector page covers the fields you can map, including a fixed event type or one read from a payload field.

Related