How do I test Datadog monitors about crash loops before a real incident?

Tares lets you test a Datadog crash loop monitor before a real incident by sending a sample payload to the source’s ingest URL, or by asking a connected agent to run test_source. You then check the label mapping in Fields and the event in Recent events. Datadog reaches Tares through the webhook connector and a Datadog webhook setup, because Tares has no connector of its own for Datadog today.

How it works

Datadog reaches Tares through the webhook connector. You create a webhook source, and Tares gives it an ingest URL. Datadog posts the JSON you write in its webhook Payload field to that URL. The payload is stored lossless, and the labels you declare map payload fields into the event.

• The payload is yours. You build it in Datadog from variables such as $ALERT_ID, $EVENT_MSG and $LINK, so a crash loop monitor sends whatever fields you put in.
• Keep it flat. event_type_field and text_template read top-level fields only.
• One primary label. Mark one label as primary. It becomes the entity key that other sources correlate on.
• Authentication. When Tares has auth on, the sender uses an Authorization Bearer header holding an ingest-scoped API key. When auth is off, no header is needed.

Because the source is a plain webhook source, you can test it without Datadog. Any HTTP POST to the ingest URL behaves like a delivery from the monitor, so you can rehearse the whole path before an incident.

Set it up

1. Create the webhook source. In the console, go to Sources → Add source → webhook. Or ask a connected agent: “Use tares to create a webhook source for my Datadog crash loop monitor.” The agent calls create_source and returns the ingest URL. See AI SRE for how sources feed the rest of Tares.
2. Declare labels for the fields your payload carries, and mark one primary. This example reads a service field and a title field:

- name: datadog-crash-loops
  connector: webhook
  config:
    text_template: '{title} on {service}'
    labels:
      - { name: service, field: service, primary: true }

3. Dry-run the mapping. Ask a connected agent to run test_source with a sample payload shaped like the one your Datadog monitor will send. The webhook page recommends this to check the label mapping first.
4. Send a sample payload to the ingest URL with curl. This is the shape of the request. Replace the body with your own crash loop fields:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "deploy", "sender": "ci", "pipeline": "web"}'

5. If Tares has auth on, add an Authorization Bearer header holding an ingest-scoped API key to your test request. Without auth, send it as shown.
6. Open the source page and check Recent events for the event. Open Fields to see which payload fields arrive and how often, and confirm the primary label holds the value you expect.
7. In Datadog, open the Webhooks integration and add a webhook whose URL is the same ingest URL. Write the Payload as flat JSON from Datadog variables. When Tares has auth on, add custom headers as JSON with an Authorization Bearer header holding the ingest-scoped API key.
8. Put @webhook- followed by your webhook name in the crash loop monitor message, so the monitor calls the webhook. When it fires, the event appears in Recent events like your test event did.

If you also run Alertmanager, its receiver has its own ingest URL and its own source.

What you get

• A checked label mapping. Fields shows which payload fields arrive and how often, so you pick a primary label that is always present before the monitor fires for real.
• A visible event. Your test payload shows up in Recent events, so you know the source stores what you send.
• A rehearsed path. The same ingest URL and the same payload shape serve the test and the Datadog webhook, so the setup you tested is the setup that runs.
• A known Tares side. You can tell whether your instance needs the Bearer header. Tares needs it only when auth is on.
• Correlation ready. The primary label becomes the entity key, so crash loop events line up with events from your other sources on the same value.

When something else fits

If your crash loop alerts already flow through Alertmanager, use the alertmanager connector instead. It makes one event per alert and keeps firing and resolved status. If you need Datadog to be the sender in every test, fire the real monitor with the webhook in its message and watch Recent events.

Related