Which labels on Datadog monitors about error spikes tie it to the right service?

Tares ties a Datadog error-spike monitor to the right service through one label that every source carries for that service, usually service, marked as the key with primary: true. Datadog reaches Tares through the webhook connector and a vendor setup, so you write the service name into the webhook payload and declare it as the primary label. Then the alert, the metrics, the logs and recent changes land on one entity timeline.

How it works

Tares has no connector of its own for Datadog today. A Datadog monitor reaches Tares through the webhook connector: Datadog posts JSON to a Tares ingest URL, and Tares stores the payload lossless.

The label is what makes the alert useful. On each source you declare labels, and the one marked primary is the key. An entity is a pair of a label and a value, such as service=checkout. A read of that entity returns every event with that label across sources, merged in time order. The concepts page covers the full model.

For an error spike, this means:

• The Datadog source carries service, read from a field in your payload.
• Your metrics, logs and deploy sources carry the same service label with the same values.
• A read of service=checkout then shows the monitor firing next to the error logs, the metric that moved and the deploy that landed minutes earlier.

The Datadog payload is a JSON object you write in the webhook Payload field, built from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so the shape is yours. Add a flat service field to it. Keep the payload flat, because event_type_field and text_template read top-level fields only, and a top-level service field is the simplest thing for a label to read.

The label name and the values both have to match. If Datadog says checkout and your logs say checkout-api, those are two entities, so settle on one spelling for each service everywhere.

Set it up

1. Create a webhook source for the monitor with service as the primary label. In the console go to Sources, Add source, webhook, or ask a connected agent to create it. The env line is optional and adds a fixed label to every event from this source.

- name: datadog-alerts
  connector: webhook
  config:
    event_type: error_spike
    labels:
      - { name: service, field: service, primary: true }
      - { name: env, const: prod }

2. Copy the ingest endpoint from the source page. Producers post to the ingest path with the ingest key that Tares generated for the source.
3. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL.
4. Write the payload as flat JSON from Datadog variables and include a service field holding the service name. When Tares has auth on, add custom headers as JSON with an Authorization Bearer header holding an ingest-scoped API key. On an instance without auth you can leave that out.
5. Put the webhook handle, which starts with @webhook- followed by the name you gave the webhook, in the monitor message so the monitor calls it.
6. Give your other sources the same label. Metrics and logs sources read service from their own fields, or use a fixed value per source. A deploy source is a second webhook source with a primary service label. Your CD pipeline posts one line per deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

7. Fire the monitor, or post a sample payload, and check Recent events and Fields on the source page. Fields shows how many sampled events carry service and its top values, so you can confirm the key is reliably present. Then open Explore, pick the service entity and read the merged timeline.

If you want a trigger on top, it watches these sources and groups by entity. Its key_field defaults to the primary label of the first source, so list the Datadog source and the others keyed by service. The AI SRE guide builds this end to end.

What you get

• One timeline per service. The Datadog alert, the metrics, the logs and recent deploys sit on one clock under the same service value.
• A reliable key. Because service is primary, key_value is set from it on every event, and selecting by service is a lookup, not a text scan.
• Triggers and agents that work per service. A firing is one dispatch for one entity, carrying that entity’s timeline across the sources, so an agent starts from correlated evidence.
• Findings on the same timeline. An agent’s conclusion is written back with the same label, so the next look at that service starts from it.
• A lossless record. The full Datadog payload is stored unmodified next to the labels.

When something else fits

• If a monitor covers several services in one alert, a single service label cannot name the right one. Send one payload per service, or pick a label that is genuinely shared, such as env or team, for that source.
• If your sources use different spellings for the same service, fix the spelling at the source first, because labels match on exact values.
• If the alerts already flow through Alertmanager, the alertmanager connector makes one event per alert and may suit you better than a webhook source.

Related