How do I test Datadog monitors about a disk filling up before a real incident?

Tares lets you test a Datadog disk monitor before an incident by posting a sample payload to the source’s ingest URL or running test_source, then checking the labels and Recent events. Datadog reaches Tares through the webhook connector and a Datadog-side setup. You rehearse the whole path with a made-up disk alert, so a real one lands with the labels you already checked.

How it works

Tares has no connector of its own for Datadog today. A Datadog monitor reaches Tares through the webhook connector: Datadog’s Webhooks integration posts JSON to a Tares ingest URL, and Tares stores the payload lossless.

• You write the payload. In Datadog you build the JSON in the webhook Payload field from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so the shape is yours.
• Labels map fields into the event. You declare a label per payload field you want to correlate on and mark one as primary. The primary label keys the entity, so a disk alert for one host lands on that host.
• Keep the payload flat. event_type_field and text_template read top-level fields only, so put the fields you need at the top level of the JSON.
• Testing is a normal POST. An ingest URL does not care whether Datadog or you sent the request, so a hand-made disk-full payload goes through the same label mapping as a real one.

This tells you whether your mapping works before a monitor fires. It does not test Datadog’s own monitor thresholds, which you set in Datadog.

Set it up

1. Create the source. In the console, go to Sources → Add source → webhook, declare a label per payload field and mark one primary. Or ask a connected agent: “Use tares to create a webhook source for my Datadog disk monitors, keyed by the payload’s host field.” The agent calls create_source and returns the ingest URL. The source below reads the event type from action, builds the text from message, and keys on host:

- name: datadog-disk
  connector: webhook
  config:
    event_type_field: action
    text_template: '{action}: {message}'
    labels:
      - { name: host, field: host, primary: true }

2. Check the label mapping first. Ask your connected agent to run test_source with a sample payload that looks like your disk alert. It shows how the labels map before anything is stored.
3. Send a sample disk alert to the ingest URL. This payload is flat and matches the source above:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "triggered", "host": "web-1", "message": "Disk space is low on /var"}'

If your Tares has auth on (tares up --auth), also send an ingest-scoped API key as a Bearer Authorization header. Without auth, the request above is enough. 4. Open the source and check Recent events. You should see one event with the event type, the rendered text and the host label. Then open Fields. It shows which payload fields arrive and how often, which helps you pick a reliable primary label. 5. Fix the mapping and repeat. Change the labels or text_template, send the sample again, and confirm the event reads the way you want. 6. Connect Datadog. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the Payload as JSON from Datadog variables, using the same top-level field names you tested. When Tares has auth on, add custom headers as JSON with an Authorization Bearer header holding an ingest-scoped API key. Then put @webhook- followed by your webhook name in the disk monitor’s message so the monitor calls it.

The AI SRE guide shows how the events from this source work with your other sources once they arrive.

What you get

• A tested path from a Datadog disk monitor to a Tares event, proven with a sample payload before any real incident.
• Confirmed label mapping. The host label, the event type and the text are visible in Recent events, and Fields shows which payload fields arrive.
• A primary label you picked on purpose, so a disk alert for one host correlates with your other sources by host.
• A payload shape you control. You change the Datadog payload and the Tares labels together, and repeat the sample post until they agree.
• A stored, lossless copy of every payload, so a field you did not map yet is still there to promote to a label later.

When something else fits

• If your disk alerts come from Prometheus rules routed through Alertmanager, the Alertmanager connector is a closer fit. It makes one event per alert and tracks firing and resolved.
• If you only want to watch Datadog-independent signals, such as metrics or logs from your own hosts, use a connector built for that source instead of a Datadog webhook.

Related