How do I test Datadog monitors about a cron job that stopped before a real incident?

Tares lets you test a Datadog monitor path before a real incident by posting a sample payload to the webhook source’s ingest URL. Then you read Recent events and Fields on the source page to confirm the event arrived and the labels mapped. Datadog reaches Tares through the webhook connector and a Datadog-side setup, and Tares has no connector of its own for Datadog today.

How it works

Datadog calls a webhook when a monitor message contains @webhook- followed by the webhook name. The body is the JSON you write in the Datadog webhook Payload field, built from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so the shape is yours.

On the Tares side, a webhook source receives that JSON at its ingest URL and stores it lossless. You declare one label per payload field you want to correlate on and mark one as primary. Two options read top-level fields only: event_type_field and text_template. Keep the payload flat so they work.

Because the source is the same whether the call comes from Datadog or from you, a hand-made POST with the same JSON shape exercises the same path a real alert will use.

For a push source, test_source takes no payload and only returns the ingest endpoint. It does not check your label mapping. The check is to post a sample payload and read the source page.

Set it up

1. Create the webhook source. Use Sources → Add source → webhook, or ask a connected agent to create a webhook source and return the ingest URL. Declare a label for each field your Datadog payload carries and mark one primary. For the labels, use the names of the fields in the payload you wrote for the cron monitor. The shape of the source looks like this:

- name: ci-events
  connector: webhook
  config:
    event_type_field: action
    text_template: '{action} by {sender}'
    labels:
      - { name: pipeline, field: pipeline, primary: true }

2. Copy the ingest endpoint from the source page.
3. Write the Datadog payload as flat JSON. Pick the fields you need from the Datadog variables, then fill in sample values in place of the variables for the test.
4. Post your sample to the ingest URL. The curl below shows the form. Replace the body with your own sample payload:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "deploy", "sender": "ci", "pipeline": "web"}'

5. On the source page, open Recent events and confirm your sample appears with the event type and text you expect. Open Fields to see which payload fields arrive and how often, and to pick a reliable key. If a label is empty, fix the field name in the label and post again.
6. If your instance is secured (tares up --auth), the sender needs an Authorization: Bearer header holding an ingest-scoped API key. Test posts need the same header. In Datadog, add it under custom headers as JSON.
7. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Use the payload you just tested. Put @webhook- followed by the webhook name in the monitor message so the monitor calls it.
8. When the monitor next fires, confirm in Recent events that the real delivery matches your test event.

What you get

• A confirmed label mapping before any real incident, so the cron job’s alert already carries the labels you want to correlate on.
• A visible event in Recent events that shows the event type and text your template produces.
• A Fields view that shows which payload fields arrive and how often.
• A source that is ready for the monitor to call, so the first stopped cron job lands in the shape you tested.

For more on how Tares uses these events once they arrive, see AI SRE.

When something else fits

• If Datadog’s payload is nested and you need deep fields in text_template or event_type_field, flatten it in the Datadog Payload field first, because those options read top-level fields only.
• If your alerts come from Alertmanager instead, use the Alertmanager connector, which fans a batch out into one event per alert and has its own ingest URL.

Related