How do I test Datadog monitors about high latency before a real incident?

Tares lets you test your Datadog high latency monitors before a real incident by sending a sample payload to the source’s ingest URL, or by running test_source with that payload. You then check the label mapping and the event on the timeline. Datadog reaches Tares through the webhook connector and a Datadog webhook setup, so the test uses the same path as a real alert.

How it works

Tares has no connector of its own for Datadog today. A Datadog monitor reaches Tares through the webhook connector and a Datadog webhook setup.

• Datadog posts a JSON object that you write in the webhook Payload field, built from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK. The shape is yours.
• Tares stores the payload lossless. The labels you declare map payload fields into the event.
• Keep the payload flat. The event_type_field and text_template options read top-level fields only.
• On a secured instance (tares up --auth), Datadog sends an Authorization Bearer header holding an ingest-scoped API key. You add it as a custom header in JSON. If Tares has auth off, no header is needed.

Because a test and a real alert use the same ingest URL and the same label mapping, a test that works tells you the real alert will land the same way. The AI SRE guide shows how these events feed correlation. If you also run Prometheus, the Alertmanager connector works the same way: you can send it a sample batch with curl to verify before an incident.

Set it up

1. Create the source. Ask a connected agent: “Use tares to create a webhook source for my Datadog latency monitor.” The agent calls create_source and returns the ingest URL. You can also use Sources → Add source → webhook and copy the ingest endpoint from the source page.

2. Declare labels for the fields your payload carries and mark one primary:

- name: ci-events
  connector: webhook
  config:
    event_type_field: action
    text_template: '{action} by {sender}'
    labels:
      - { name: pipeline, field: pipeline, primary: true }

Use your own source name and your own payload fields in place of these.

3. Check the mapping first. Run test_source with a sample payload, shaped like the JSON you will write in Datadog. It shows how Tares maps your labels before anything is sent.

4. Send a sample payload to the ingest URL. Replace the fields with the ones in your Datadog payload:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "high latency", "sender": "datadog", "pipeline": "web"}'

On a secured instance, add an Authorization: Bearer header with an ingest-scoped API key to the request.

5. Open the source and check Recent events for the new event. Fields shows which payload fields arrive and how often, so you can confirm the primary label and pick a reliable key.

6. Wire up Datadog. Open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as flat JSON from Datadog variables. When Tares has auth on, add the custom header as JSON with the Authorization Bearer header. Then put @webhook- followed by your webhook name in the latency monitor message so the monitor calls it.

7. Repeat the check with the real payload. Use the Datadog webhook test or a monitor test notification, if you use one, and confirm the event shows the same labels and text as your curl test.

What you get

• Proof that the ingest URL, the authentication and the label mapping work before a latency incident.
• A visible event on the timeline with the labels and text you chose, so you see what an on-call agent or trigger will see.
• A primary label that keys the event to the right entity, so it correlates with your other sources.
• Early warning about a payload mistake, such as a field the labels do not match or a nested field that text_template cannot read.
• A repeatable check. You can rerun the same curl after you change the payload or the labels.

When something else fits

• If your latency alerts already route through Prometheus Alertmanager, point a receiver at an Alertmanager source. It makes one event per alert and tracks firing and resolved.
• If you want to verify only the label mapping and not Datadog delivery, test_source alone is enough. Send the real monitor notification when you need to check the full path from Datadog.

Related