Which labels on Datadog monitors about failed deploys tie it to the right service?

Tares ties a Datadog monitor to the right service through one label that every source carries with the same value, usually service, marked as the primary label (the key). Put service in the Datadog webhook payload and declare it as the key on your alert, metrics, logs and deploy sources. The alert and the evidence around it then land on one entity timeline.

How it works

Tares has no connector of its own for Datadog today. Datadog monitors reach Tares through the webhook connector and a Datadog-side webhook setup. See Inbound webhook for the connector itself.

A label is a named axis on an event. The label marked as primary is the key, and its value is stored as the event’s key value. An entity is a pair of a label and a value, for example service and checkout. Because labels are indexed, reading service: checkout is a lookup, not a text scan. The read returns that service’s alerts, metrics, logs and deploys merged in time order. The selector is a strict AND, so adding a second label narrows the result. Read more in Concepts.

For failed deploys, choose service as the shared label:

• The Datadog webhook source carries service from the payload you write.
• Your metrics source reads service from the metric labels.
• Your logs source gets service from a constant or from your OpenTelemetry resource attributes.
• Your deploy webhook source reads service from the deploy payload.

The values must match exactly across sources. Checkout and checkout-api are two different entities, so use one spelling everywhere.

Set it up

1. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as flat JSON built from Datadog variables such as $EVENT_MSG. Add a service field with the service name, one webhook per service.
2. If Tares has auth on, add custom headers as JSON in Datadog with an Authorization Bearer header holding an ingest-scoped API key. On an open instance you do not need it.
3. Put @webhook- followed by your webhook’s name in the monitor message, so the monitor calls it.
4. In Tares, create a webhook source for the Datadog alerts. Declare a service label and mark it primary. Keep the payload flat, because event_type_field and text_template read top-level fields only.

- name: datadog-alerts
  connector: webhook
  config:
    event_type: monitor_alert
    text_template: '{message}'
    labels:
      - { name: service, field: service, primary: true }

5. Post a sample payload to the source’s ingest URL to check the mapping.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"service": "checkout", "message": "deploy failed"}'

6. Give your other sources the same service key. A metrics source can read it from the metric labels. A logs source can set it as a constant:

labels:
  - { name: service, field: metric.service, primary: true }

labels:
  - { name: service, const: api-server, primary: true }

7. For recent changes, add a second webhook source with event_type deploy and a primary service label. Have your CD pipeline post to it on every deploy. The AI SRE guide shows this deploy source in full.
8. Open the source page and check Recent events and Fields. Fields reports how many sampled events carry each label and its top values, so you can confirm service is reliably present before you rely on it as the key.

What you get

• One timeline per service. When a Datadog monitor about a failed deploy fires, the alert sits next to that service’s metrics, logs and deploy events on one clock.
• A read of service: checkout returns all of it in a single response, and each row shows the labels it matched on.
• Triggers keyed by service. A trigger watching these sources groups by the primary label, so one firing is one service, carrying that service’s timeline.
• A finding written by an agent lands on the same entity, so the next reader starts from the previous conclusion.
• A deploy event landing shortly before the alert is visible in the same read, so the likely cause is in front of you.

When something else fits

• If your monitors carry no service value you can put in the payload, declare a constant label on the Datadog source instead, with one source per service. That is a good fit when a monitor belongs to exactly one service.
• If a different label such as repo or pipeline is the one every source already shares, mark that one as the key instead. The mechanism is the same.

Related