How do I tie Datadog monitors to the right service?

In Tares, you tie a Datadog monitor to a service by mapping one field of the alert payload to the primary label that your metrics and logs also carry. Alerts then land on the same per-service timeline as that service’s other signals. Datadog reaches Tares through the webhook connector and a vendor setup.

How it works

Tares builds one timeline per entity from the labels on each source. When a metrics source, a logs source and your Datadog source all carry the same primary label, for example service, their events join on one clock. The AI SRE guide builds this pattern end to end, with every source keyed by the same service label.

Datadog reaches Tares through the webhook connector and a vendor setup. Tares has no connector of its own for Datadog today. Datadog posts the payload to a Tares ingest URL, and Tares stores it lossless.

• You write the payload in the Datadog webhook Payload field as JSON. You can use Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so the shape is yours.
• Include one field that holds the service name. A label on your Tares source maps that field to the primary label.
• Keep the payload flat. event_type_field and text_template read top-level fields only.
• On a secured instance, the Authorization Bearer header with an ingest-scoped API key is the credential.

Set it up

1. Create a webhook source in Tares. Go to Sources, Add source, webhook, or ask a connected agent to create it. Declare a label for the field that carries the service name and mark it primary. Use the same label name your metrics and logs sources use.

- name: datadog-monitors
  connector: webhook
  config:
    event_type_field: action
    text_template: '{action}: {message}'
    labels:
      - { name: service, field: service, primary: true }

2. Save the source and copy the ingest endpoint from the source page. When Tares has auth on, creating the source in the console mints an ingest-scoped API key and shows it once.

3. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL.

4. Write the payload as flat JSON, with a service field that holds the service name, plus action and message fields to match the source above. When Tares has auth on, add custom headers as JSON with an Authorization header whose value is Bearer followed by your ingest-scoped API key.

5. Put @webhook- followed by your webhook name in the monitor message, so the monitor calls the webhook.

6. Check the mapping before you rely on it. Post a sample payload to the ingest URL, or ask a connected agent to run test_source with a sample payload.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"service": "checkout", "action": "triggered", "message": "High error rate"}'

7. Open Recent events on the source page to see the event. Fields shows which payload fields arrive and how often, which helps you pick a reliable key. Then open Explore and pick the service. The Datadog event sits on the same timeline as the metrics and logs.

What you get

• Every Datadog monitor event lands on the timeline of the service it names, next to that service’s metrics and logs.
• One correlated read per service. A trigger or an agent works from the alert and the evidence around it, not from the alert alone.
• A payload you control. You decide which fields Datadog sends, so you choose the label that identifies the service.
• A lossless record. Tares stores the full payload, so fields you did not map to labels remain available.
• A readable event line, from text_template and the event type from event_type_field, both taken from top-level fields.

When something else fits

• If the same service name is not available in the monitor, the join has nothing to match on. Add the service name to the payload first, or the event will not reach the right timeline.
• If your alerts already flow through Prometheus rules, the prometheus_alerts or alertmanager connectors take those alerts directly and need no Datadog webhook.
• If you need a payload with nested fields as labels, flatten it in the Datadog payload first. Keep event_type_field and text_template pointed at top-level fields.

Related