Which labels on Datadog monitors about high latency tie it to the right service?

Tares ties a Datadog high latency monitor to its service through a service label that you mark as primary, the key, on the webhook source. Every other source you watch for that service carries the same service label. The alert, the metrics, the logs and recent deploys then land on one entity timeline for that service.

How it works

Datadog reaches Tares through the webhook connector and a Datadog-side setup. Tares has no connector of its own for Datadog today. The webhook connector stores the payload lossless and maps payload fields into labels that you declare.

• The payload is yours. In Datadog you write the webhook payload as JSON, built from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK. Add a service field to it so the alert says which service it is about.
• Keep it flat. event_type_field and text_template read top-level fields only, so put service at the top level of the payload.
• The primary label is the key. The label marked primary: true sets key_value on every event. An entity is a label and value pair, such as service and checkout.
• Same label everywhere. A read with the selector service: checkout returns every source’s events for that service, merged in time order. This works only if each source carries the label under the same name.

The selector is a strict AND, so adding a second label, such as env, narrows the timeline. See Concepts for how labels, keys and entities fit together.

Set it up

1. Create a webhook source for the monitors. Declare service as the primary label, and add an env label if you want to narrow reads by environment:

- name: datadog-monitors
  connector: webhook
  config:
    labels:
      - { name: service, field: service, primary: true }
      - { name: env, const: prod }

2. Copy the ingest URL from the source page. Post a sample payload and check that the label arrives:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"service": "checkout"}'

3. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as flat JSON from Datadog variables, and include the service field.
4. If Tares has auth on, add custom headers as JSON in Datadog with an Authorization Bearer header holding an ingest-scoped API key. On a secured instance that header is the credential.
5. Put @webhook- followed by your webhook name in the high latency monitor message, so the monitor calls the webhook.
6. Give your other sources the same service label. Metrics from Prometheus read it from the metric labels. Logs and alerts get it from a label on their sources, and a second webhook source carries deploys with a primary service label read from the payload’s service field. The Build an AI SRE guide walks through this wiring end to end.
7. Open the source’s Fields tab and check the coverage of service. It shows how many sampled events carry the label, so you can confirm the key is reliably present. The webhook connector page covers the options in full.

What you get

• One timeline per service. The Datadog alert sits next to the metric that moved, the log lines around it and the deploy that landed before it.
• A reliable key. key_value is set from the service label on every event, and the Fields tab shows coverage so you can see gaps.
• Reads that are lookups, not text scans, because labels are indexed. Reading service: checkout returns that service’s alerts, logs, metrics and deploys in one response.
• Triggers keyed by service. A trigger groups its condition per entity, so a latency alert for one service wakes an agent with that service’s timeline only.
• Findings that stay put. An agent’s conclusion is written back with the same label, so the next read of that service starts from it.

When something else fits

• If the monitors come from Prometheus rules, the prometheus_alerts or alertmanager connectors take the alerts directly, and you map service from their labels instead of a hand-written Datadog payload.
• If a monitor covers several services, send one webhook per service or write the payload so each posts a single service value. One label value maps to one entity.

Related