Which labels on Datadog monitors about 5xx errors on an API tie it to the right service?

Tares ties Datadog 5xx monitors to the right service through one label, service, that every source carries with the same value and that you mark as the primary label (the key). Tares has no connector of its own for Datadog today, so the monitor reaches Tares through the webhook connector and a Datadog webhook setup. Once the alert, the metrics, the logs and your deploys all carry the same service value, one read returns them on a single timeline.

How it works

Every event carries labels, and a label is a named axis. The label you mark primary: true is the key, and its value is stored as the event’s key_value. An entity is a (label, value) pair, such as (service, checkout). A read with the selector { service: "checkout" } returns every event with that label from every source, merged in time order. See Concepts for the full model.

For 5xx errors on an API, pick the label that already names the service in your other sources and reuse it exactly:

• service is the key. It must have the same name and the same spelling of the value on every source, for example checkout everywhere, not checkout in one source and checkout-api in another.
• env, as an optional second label, narrows a read to one environment. A fixed value such as env set to prod can be declared with const.

A Datadog monitor reaches Tares as a webhook delivery. You write the JSON payload in Datadog’s webhook Payload field, so the shape is yours. Put a top-level service field in it, carrying the service name, and declare a label that reads it. Keep the payload flat, because event_type_field and text_template read top-level fields only. The full payload is stored lossless, so nothing you send is dropped.

A trigger can then watch these sources together. Its key_field is optional and defaults to the primary label of the first source, so keying every source by service keeps one firing per service.

Set it up

1. Choose the key. Use service, and decide the exact value for your API, for example checkout. Use the same value in the metrics, logs, alert and deploy sources.

2. Create a webhook source for Datadog. In the console go to Sources, Add source, webhook, or use POST /api/sources with the same object. Save it and copy the ingest endpoint from the source page.

- name: datadog-alerts
  connector: webhook
  config:
    labels:
      - { name: service, field: service, primary: true }
      - { name: env, const: prod }

3. Add the webhook in Datadog. Open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as flat JSON from Datadog variables, with a service field holding your service name. When Tares has auth on, add custom headers as JSON with an Authorization Bearer header holding an ingest-scoped API key. The key is needed only when Tares has auth on.

4. Call the webhook from the monitor. In the 5xx monitor’s message, put the @webhook- handle followed by the name you gave the webhook in step 3.

5. Give the other sources the same label. The metrics source reads service from the metric labels. A logs source can set it as a constant, as in this example:

- name: logs
  connector: docker_logs
  poll: 5s
  config:
    container: tares-demo-api-server   # the container name (docker ps)
    labels:
      - { name: service, const: api-server, primary: true }

6. Add recent changes. Create a second webhook source with a primary service label read from the payload’s service field, then post one request from your deploy pipeline on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

7. Check the key. Post a test alert, then open the source’s Fields tab. It shows how many sampled events carry service and its top values, so you can confirm the key is reliably present. Then open Explore and pick the service to see the merged timeline. The webhook connector page covers the ingest endpoint and the Recent events view.

What you get

• One timeline per service. When the Datadog 5xx monitor fires, the alert lands next to the metrics, the logs and the latest deploy for the same service, on one clock.
• A lookup instead of a text search. Labels are indexed, so selecting by { service: "checkout" } is a lookup.
• Findings on the same entity. An agent that diagnoses the alert writes its finding onto the service’s timeline, so the next investigation starts from that conclusion.
• Deploy and config changes in the same read, so a pattern such as a deploy followed by a jump in 5xx is visible in one response.

The AI SRE guide shows the same pattern end to end: sources keyed by service, a trigger over them and an agent that writes back.

When something else fits

• If your Datadog payload cannot carry the service name, set a constant label on a dedicated webhook source per service instead. Each source then stamps its own service value.
• If a source has no reliable service value, keep it as a secondary label and key on the one that is always present. The Fields tab shows coverage for each label.

Related