Which labels on Datadog monitors about slow database queries tie it to the right service?

Tares ties a Datadog slow-query monitor to the right service through one label, service, that you mark as the primary label (the key) on every source. Use the same label name and the same value in the Datadog webhook source, your metrics, your logs and your deploy events. The alert, the metrics, the logs and recent changes then land on one entity timeline.

How it works

• Every event carries labels. An entity is a (label, value) pair, such as service=checkout. The label you mark primary: true is the key, and Tares stores its value as key_value on each event.
• Datadog reaches Tares through the webhook connector and a vendor setup in Datadog. Tares has no connector of its own for Datadog today. You write the webhook payload as JSON, so the service field in it is yours to include.
• The label name and value must match across sources. A read returns every event for one entity, merged in time order. If the Datadog alert says service=checkout and your metrics, logs and deploys also say service=checkout, one read returns all of them together.
• Reads are a strict AND of label selectors. service=checkout returns that service’s alerts, metrics, logs and deploys. Adding a second label, such as env set with const: prod, narrows the result.
• A trigger’s key_field is optional. When you leave it out, the trigger uses the primary label of the first source it watches, so a primary service label on that source is enough.

Set it up

1. Create a webhook source for the Datadog alerts. Declare a service label that reads the payload’s service field and mark it primary:

- name: datadog-alerts
  connector: webhook
  config:
    labels:
      - { name: service, field: service, primary: true }

2. Copy the ingest URL from the source page. In Datadog, open the Webhooks integration and add a webhook whose URL is that Tares ingest URL.
3. Write the payload as JSON from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK. Add a top-level service field holding the name of the service that issues the slow queries. Keep the payload flat, because event_type_field and text_template read top-level fields only.
4. If Tares has auth on, add custom headers as JSON in the Datadog webhook, with an Authorization Bearer header that holds an ingest-scoped API key. On an instance without auth the header is not needed.
5. Put @webhook- followed by your webhook’s name in the slow-query monitor message, so the monitor calls it.
6. Give your other sources the same primary label. Metrics, logs and alerts each get a service label with the same values as the Datadog payload. For recent changes, create a second webhook source with a primary service label and have your pipeline post a deploy on every release:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

7. Check the key. Post a test alert, then open the source’s Fields tab. It shows how many sampled events carry service and the top values. Then open Explore, pick the service and confirm the alert sits next to the metrics, logs and deploys. The concepts page defines labels, keys and entities in full, and the AI SRE guide shows the same per-service wiring end to end.

What you get

• A slow-query alert from Datadog appears on the same timeline as that service’s metrics, logs and deploys, on one clock.
• A deploy such as a raised database pool size sits right before the alert, so an agent or a person sees the likely change first.
• A trigger over those sources fires once per service, and the agent it wakes gets that service’s correlated timeline.
• Findings an agent writes back carry the same label, so the next read of the service starts from the previous conclusion.
• A coverage check in the Fields tab tells you early if the key is missing on some events.

When something else fits

• If you cannot put a service name in the Datadog payload, create one webhook source per service and give it a const service label, which applies a fixed value to every event from that source.
• If your sources already agree on another label, such as repo or a team name, mark that one primary instead. The rule is the same: one label every source carries, with matching values.

Related