Which labels on Datadog monitors about crash loops tie it to the right service?

Tares ties a Datadog crash loop monitor to the right service through a service label, the same label your metrics, logs and deploy sources carry, marked as the key with primary: true. Datadog reaches Tares through the webhook connector and a Datadog setup, and you write the service name into the payload yourself. Every event with service set to the same value then lands on one entity timeline.

How it works

Tares groups events into entities. An entity is a label and value pair such as service and checkout. Each source declares which payload field fills which label, and the label marked primary is the key. Its value is stored as the event’s key_value. See Concepts for the full model.

Datadog reaches Tares through the webhook connector and a vendor setup. Tares has no connector of its own for Datadog today. A Datadog webhook posts a JSON object that you write in the webhook Payload field, built from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so the shape is yours. That gives you control over the label:

• The key label is service. Choose the one label that every source you use for this service already carries. Name it the same everywhere, for example service, so a read for service checkout returns the crash loop alert, the metrics, the logs and the recent deploys merged in time order.
• Put the service name in the payload as a top-level field. Keep the payload flat, because event_type_field and text_template read top-level fields only.
• Add a second label only if it narrows the read. A read is a strict AND over the labels you pass, so a label such as env, set with const, lets you narrow a service’s timeline to one environment.

The webhook connector stores the original payload unmodified, so nothing you send is lost even if you do not map it to a label.

Set it up

1. Pick the label name every source of this service already uses. If your metrics, logs and deploy sources use service, use service here too.

2. Create a webhook source with a label for the payload field that carries the service name, marked primary. In the console, go to Sources, then Add source, then webhook. You can also ask a connected agent to create it.

- name: datadog-monitors
  connector: webhook
  config:
    labels:
      - { name: service, field: service, primary: true }
      - { name: env, const: prod }

3. Copy the ingest endpoint from the source page. It is the URL Datadog will call.

4. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the Payload as flat JSON that includes a top-level service field holding the service name, plus the Datadog variables you want, such as $EVENT_MSG and $LINK.

5. If Tares has auth on, add custom headers as JSON in the Datadog webhook with an Authorization Bearer header holding an ingest-scoped API key. On a secured instance that header is the credential.

6. In the crash loop monitor’s message, put @webhook- followed by the name of your Datadog webhook, so the monitor calls it.

7. Fire a test notification, then open the source page in the console. Recent events shows what arrived, and Fields shows how many sampled events carry service. If coverage is below full, fix the payload before you rely on the key.

If you run Tares locally, start it with this command and open the console at http://127.0.0.1:8787:

tares up

If you add a trigger over these sources, its key_field defaults to the primary label of the first source. Set key_field to service if the first source is not the Datadog one. The AI SRE guide shows the same pattern end to end: sources keyed by one service label, one trigger, one agent.

What you get

• A crash loop alert from Datadog that sits on the same timeline as the metrics, logs and deploys for that service, merged on one clock.
• One read, with service as the selector, that returns all of it. Each row carries the labels it matched on, so the read is self-describing.
• An agent woken by a trigger that starts from the whole correlated timeline for the right service, not from the alert text alone.
• A findings trail. An agent’s conclusion is written back with the same label, so the next investigation of that service starts from it.
• A lossless copy of the Datadog payload, so you can add a label later without losing earlier events.

When something else fits

• If your alerts already reach Tares through a source that carries the service label, you do not need a separate Datadog webhook source. Keep one label name across all sources instead.
• If a Datadog monitor cannot put a service value into the payload, the key will be missing. Check Fields coverage first, and fix the payload or pick another label that every event does carry.

Related