How do I test Datadog monitors about slow database queries before a real incident?

Tares lets you test Datadog slow-query monitors before an incident by posting a sample payload to the webhook source’s ingest URL, or by running test_source through a connected agent. You then check the label mapping and the event under Recent events. Datadog reaches Tares through the webhook connector and a Datadog-side webhook setup, so the test uses the same path as a real alert.

How it works

Tares has no connector of its own for Datadog today. Datadog reaches Tares through the webhook connector and a Datadog webhook setup. The monitor calls a Datadog webhook, and the webhook posts a JSON object to the ingest URL of a webhook source in Tares.

You write the payload in Datadog’s webhook Payload field from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so the shape is yours. Tares stores the payload lossless and maps the fields you declare as labels into the event.

Because the source accepts any JSON, you can post the same payload by hand before any monitor fires. That checks three things:

• The ingest URL accepts the request.
• Your labels pick up the right fields from the slow-query payload.
• The event looks the way you want on the Recent events list.

Keep the payload flat. The event_type_field and text_template options read top-level fields only.

Set it up

1. Create a webhook source for the monitors. In the console, go to Sources → Add source → webhook. You can also ask a connected agent to create a webhook source and give you the ingest URL. Declare a label for each payload field you want to correlate on, and mark one as primary. This is the shape to start from:

- name: ci-events
  connector: webhook
  config:
    event_type_field: action
    text_template: '{action} by {sender}'
    labels:
      - { name: pipeline, field: pipeline, primary: true }

Replace the field names with the top-level fields your Datadog payload carries, for example the monitor name or the database service. Copy the ingest endpoint from the source page.

2. Check the mapping with a sample payload. Ask a connected agent to run test_source with a sample payload, or post one yourself:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "deploy", "sender": "ci", "pipeline": "web"}'

Put the JSON you wrote in the Datadog Payload field in the body, so the test matches what Datadog sends.

3. If your Tares instance has auth on, send an ingest-scoped API key as a Bearer Authorization header with the request. Without auth you need no header.

4. Open the source and check Recent events. Confirm the event type and text read the way you expect. Open Fields to see which payload fields arrive and how often, and make sure your primary label is a field that is always present.

5. Wire up Datadog. Open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as JSON from Datadog variables. When Tares has auth on, add custom headers as JSON with an Authorization Bearer header holding the ingest-scoped API key. Put @webhook- followed by your webhook name in the slow-query monitor’s message so the monitor calls it.

6. Fix any mismatch between the test event and what you want, adjust the labels or the template, and send the sample again. Repeat until the event is right.

What you get

• You know the ingest URL, the key and the payload shape work before a slow-query monitor fires for real.
• Your labels and primary key are checked against the actual fields on Fields, so the event correlates with your other sources on the first real alert.
• The event type and text from your template show up on Recent events the way responders will read them.
• Tares keeps the full payload, so a field you did not label is still there to promote later.

With that in place, the AI SRE workflow can correlate a slow-query alert with the rest of what Tares sees from the first alert.

When something else fits

• If your Datadog alerts already flow through Prometheus Alertmanager, the alertmanager connector turns each alert in a batch into its own event, and it has its own test batch you can send with curl.
• If you need to keep the payload nested, flatten it before it reaches Tares, because the event type field and text template read top-level fields only.

Related