How do I test Datadog monitors about error spikes before a real incident?

Tares lets you test Datadog error-spike monitors before a real incident by sending a sample payload to the source’s ingest URL, or by running test_source with that payload. Datadog reaches Tares through the webhook connector and a Datadog webhook setup. You then check the label mapping and the event under Recent events and Fields. Once the sample looks right, you point the Datadog monitor at the same URL.

How it works

Tares has no connector of its own for Datadog today. Datadog reaches Tares through the webhook connector and a Datadog-side webhook setup.

• You create a webhook source in Tares. It gives you an ingest endpoint.
• Datadog posts a JSON object to that endpoint. You write the payload yourself in the webhook Payload field from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so the shape is yours.
• Tares stores the payload lossless. The labels you declare map payload fields into the event so Tares can correlate it with your other sources.
• Because the payload is your own, you can send the same shape by hand first. That is the test: a sample payload that looks like your real monitor alert, posted before any monitor fires.

The webhook connector reads event_type_field and text_template from top-level payload fields only. Keep the Datadog payload flat so both work.

Set it up

1. Create the webhook source. Ask a connected agent to create a webhook source for your Datadog monitors, or use Sources, Add source, webhook in the console. Declare one label per payload field you want to correlate on, and mark one as primary. The example below is the shape of a webhook source. Swap the field names for the ones your Datadog payload carries.

- name: ci-events
  connector: webhook
  config:
    event_type_field: action
    text_template: '{action} by {sender}'
    labels:
      - { name: pipeline, field: pipeline, primary: true }

2. Copy the ingest endpoint from the source page. The response from POST /api/sources also includes the ingest_key.

3. Send a sample payload. Post a JSON body that mirrors what your Datadog webhook will send, with sample values in place of the Datadog variables for your error-spike monitor. The command below shows the call. Replace the body with your own sample.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "deploy", "sender": "ci", "pipeline": "web"}'

4. If your Tares instance has auth on, add an Authorization: Bearer header holding an ingest-scoped API key to the request. The same header is what Datadog will send later.

5. Or skip curl. Ask your connected agent to run test_source with your sample payload. It checks the label mapping for you before you send anything to the live source.

6. Check the result. Open the source page and look at Recent events for the new event. Open Fields to see which payload fields arrive and how often. Confirm the primary label holds the value you want to key on, such as a service name, and that the event type and text read the way you expect. Adjust the labels and repeat until they do.

7. Connect Datadog. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as JSON from Datadog variables. When Tares has auth on, add custom headers as JSON with an Authorization Bearer header holding an ingest-scoped API key. Then put @webhook- followed by your webhook name in the error-spike monitor message so the monitor calls it.

If you also run Alertmanager, the alertmanager connector uses the same check: send a test batch to its ingest endpoint and read Recent events.

What you get

• You see how a Datadog error-spike alert will look in Tares before a real one fires.
• You confirm the label mapping, the primary label, the event type and the text template with a sample payload.
• You find payload problems early, such as a nested field that event_type_field or text_template cannot read, while nothing is on fire.
• You confirm that the Authorization Bearer header works on a secured instance.
• When the monitor does fire, the event lands on a source you have already checked, and Tares can correlate it with your other sources.

When something else fits

• If your alerts come from Prometheus through Alertmanager, use the alertmanager connector. It fans a batch out into one event per alert and has its own ingest URL.
• If your monitor sends a nested payload that you cannot flatten in Datadog, expect to send a flat shape instead, because the event type and text template read top-level fields only.
• For any other sender of JSON over HTTP, the webhook connector page covers the same setup and the same checks.

Related