What should Datadog monitors about slow database queries include for an AI SRE?

Tares works best when a Datadog monitor about slow database queries sends a service name, a severity, a summary and a start time, plus the alert link and ID. Datadog reaches Tares through the webhook connector and a Datadog-side setup, because Tares has no connector of its own for Datadog today. The service name matters most: it lets the agent join the alert to the metrics, logs and deploys already on that service’s timeline.

How it works

A Datadog webhook posts a JSON payload that you write yourself, so the shape is yours. Tares stores the payload lossless in a webhook source. Labels you declare map payload fields into the event, and one label is marked primary.

For slow database queries, send these fields:

• Service name. Use the same value your other sources carry in their service label. Tares builds one timeline per service, so an alert with a matching service lands next to the metrics and logs for that service. This is what lets the agent tie the alert to evidence. Without it the alert sits alone.
• Severity. Lets a trigger or the agent separate a warning from a page.
• Summary. The monitor message, from the Datadog variable $EVENT_MSG. Name the database or the query path and the threshold that tripped, so the text reads as a finding on its own.
• Start time. An ISO-8601 timestamp for when the alert began. Point event_time_field at it, so the event sits at the moment the slowdown started and not at the moment Tares received it. The agent can then line it up with a deploy or a config change.
• Link and alert ID. Use $LINK and $ALERT_ID, so a person or the agent can go back to the monitor.

Keep the payload flat. event_type_field and text_template read top-level fields only, so a nested object cannot feed them.

The webhook is the only link between the two products. Tares does not read Datadog on its own, so any field the agent should see has to be in the payload.

Set it up

1. In Tares, create a webhook source (Sources, Add source, webhook) or ask a connected agent to create one. Declare a label per field you want to correlate on and mark service as primary. Save, then copy the ingest endpoint from the source page.

- name: datadog-slow-queries
  connector: webhook
  config:
    event_type_field: action
    text_template: '{summary}'
    event_time_field: started_at
    labels:
      - { name: service, field: service, primary: true }
      - { name: severity, field: severity }

The field names are the ones you choose in your Datadog payload. action can hold a fixed word such as the monitor state, so event_type_field has something to read.

2. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL.
3. Write the payload as flat JSON from Datadog variables. Put the service, severity, summary, start time, link and alert ID in top-level fields that match the names in your source.
4. If Tares runs with auth on, add custom headers as JSON in the Datadog webhook with an Authorization Bearer header that holds an ingest-scoped API key. Creating a push source in the Tares console mints that key and shows it once. If auth is off, leave the header out.
5. Put @webhook- followed by your webhook name in the slow-query monitor message, so the monitor calls the webhook.
6. Trigger the monitor, or ask a connected agent to run test_source with a sample payload. Then open Recent events on the source and check Fields. It shows which payload fields arrive and how often, so you can confirm the service and start time come through every time.

From there, follow the AI SRE guide to add a trigger over the sources keyed by service and to choose who gets woken.

What you get

• Each slow-query alert lands as an event on the right service’s timeline, merged on one clock with that service’s metrics, logs and deploys.
• The agent reads a summary that already names the database or query path, so its first look starts from the monitor’s own words.
• The start time you send places the alert at the moment the slowdown began, so the agent can compare it with a deploy or config change and say “deploy landed, queries slowed minutes later”.
• Severity and the alert link travel with the event, so a finding can point a person back to the Datadog monitor.
• Your Datadog monitors keep owning the alerting. Tares ingests what fired, correlates it and handles the waking.

When something else fits

If your slow-query signal already lives in Prometheus, the Prometheus alerts connector is a closer fit. Tares polls it and builds the payload for you, so there is nothing to design. If you cannot add the service name to the Datadog payload, fix that first, because the alert will not join a service timeline without it.

Related