How do I test Datadog monitors about failed deploys before a real incident?

Tares lets you test a Datadog failed-deploy monitor before a real incident by sending a sample payload to the source’s ingest URL or by running test_source with that payload. Datadog reaches Tares through the webhook connector and a Datadog-side webhook setup. Tares has no connector of its own for Datadog today. After the test, you check the label mapping and the event under Recent events.

How it works

Datadog calls a webhook you define, and you write its JSON payload from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so the payload shape is yours. On the Tares side that call lands on a webhook source, which stores the payload lossless and maps payload fields into labels you declare.

Three points shape the test:

• Declare one label per payload field you want to correlate on, and mark one primary.
• Keep the payload flat. event_type_field and text_template read top-level fields only.
• The source page lists the payload fields that arrive and how often under Fields, so you can see which ones are reliable keys.

A failed-deploy monitor is an ordinary Datadog monitor that calls this webhook, so you can send your own sample payload to the same ingest URL ahead of time.

Set it up

1. Create the source. In the console go to Sources → Add source → webhook, or ask a connected agent to create a webhook source for your Datadog events. Declare a label for each field your Datadog payload carries and mark one primary. The shape looks like this:

- name: ci-events
  connector: webhook
  config:
    event_type_field: action
    text_template: '{action} by {sender}'
    labels:
      - { name: pipeline, field: pipeline, primary: true }

Replace the field names with the top-level fields of the JSON you plan to write in Datadog’s Payload field. Copy the ingest endpoint from the source page.

2. Check the label mapping first. Ask your connected agent to run test_source with a sample payload shaped like your Datadog payload. Adjust the labels until the mapping is what you want.

3. Send a sample payload to the ingest URL. This example posts one object:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "deploy", "sender": "ci", "pipeline": "web"}'

If your instance has auth on, the sender must include an Authorization Bearer header holding an ingest-scoped API key. Creating a push source in the console mints one for the source and shows it once.

4. Open the source page. Look for your event under Recent events, and open Fields to confirm your labels and the event type match what you intended. If the event type or text looks wrong, fix event_type_field or text_template and send the sample again.

5. Wire up Datadog. In the Webhooks integration, add a webhook whose URL is the Tares ingest URL. Write the payload as flat JSON from Datadog variables. If Tares has auth on, add custom headers as JSON with an Authorization Bearer header holding the ingest-scoped API key. Then put the @webhook mention for your webhook’s name in the failed-deploy monitor message so the monitor calls it.

6. Check again when the monitor next calls the webhook. Open Recent events and Fields and compare the event with the one from your sample. If a field you rely on is missing from the sample, add it to the sample or to the Datadog payload before you rely on the monitor.

What you get

• A webhook source whose labels you have already checked with a sample payload, with one primary label to key the entity.
• An event for a failed deploy visible under Recent events, plus a Fields view that shows which payload fields arrive and how often.
• A Datadog monitor that is already connected to a working ingest URL, so the first real alert is not also the first time you look at the mapping.
• A flat payload that works with event_type_field and text_template.

For the wider picture of how events from a source like this are used, see AI SRE.

When something else fits

• If your Datadog alerts already pass through Alertmanager, the alertmanager connector fans each alert into its own event and has its own ingest URL.
• If you only need to see that a payload parses, a single curl post to the ingest URL is enough. Use test_source when you want the label mapping checked first.

Related