Skip to Content

How do I test Datadog monitors about a disk filling up before a real incident?

Tares lets you test a Datadog disk monitor before a real incident by posting a sample payload to the ingest URL of your webhook source. You then check the label mapping in Recent events and Fields on the source page. Datadog reaches Tares through the webhook connector and a vendor setup, because Tares has no connector of its own for Datadog today.

How it works

Datadog sends monitor notifications to Tares through its Webhooks integration, and Tares receives them with a webhook source.

  • You write the payload yourself in the Datadog webhook Payload field, as JSON built from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK. The shape is yours, so the same shape is what you post by hand to test.
  • Tares stores the payload lossless and maps the fields you declare as labels into the event. One label is primary and keys the entity.
  • Keep the payload flat. event_type_field and text_template read top-level fields only.
  • For a push source, test_source takes no payload and only returns the ingest endpoint, so it does not check your label mapping. The check is a sample payload posted to the ingest URL, read back on the source page.
  • If Tares has auth on, the credential is an Authorization Bearer header holding an ingest-scoped API key. Without auth, no header is needed.

This fits the wider AI SRE flow. The disk alert becomes an event Tares correlates with your other sources.

Set it up

  1. In Tares, open Sources, choose Add source, then webhook. Declare a label for each field your Datadog payload carries, for example the host or mount point of the disk, and mark one as primary. Save, then copy the ingest endpoint from the source page. An agent can do the same with create_source, which returns the ingest URL.

    The example below shows the shape of a webhook source. Swap the field names for the ones in your disk payload.

    - name: ci-events connector: webhook config: event_type_field: action text_template: '{action} by {sender}' labels: - { name: pipeline, field: pipeline, primary: true }
  2. Post a sample payload to the ingest URL. Use the same flat JSON object your Datadog Payload field will produce, filled with realistic values for a filling disk. The command below is the pattern, and you replace the body with your own payload:

    curl -X POST http://localhost:8787/ingest/<ingest_key> \ -H 'Content-Type: application/json' \ -d '{"action": "deploy", "sender": "ci", "pipeline": "web"}'

    If Tares has auth on, add an Authorization Bearer header holding an ingest-scoped API key to the request.

  3. Open the source page and read Recent events. Confirm that your sample arrived as an event and that the event type and text look right.

  4. Read Fields on the same page. It shows which payload fields arrive and how often, so you can confirm your labels point at fields that exist and pick a reliable primary label. Adjust the labels and post again until the mapping is right.

  5. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as JSON from Datadog variables, matching the sample you tested. When Tares has auth on, add custom headers as JSON with an Authorization Bearer header holding the ingest-scoped API key.

  6. Put @webhook- followed by your webhook name in the disk monitor message so the monitor calls it. When the monitor fires, check Recent events again and compare the real event with your sample.

What you get

  • A label mapping you checked against a sample before any real incident, so the primary label and the other labels are already right when the disk monitor fires.
  • A visible event in Recent events that shows exactly how Tares reads your disk payload.
  • A Fields view that tells you which payload fields arrive, so you catch a mismatch between your Datadog payload and your labels early.
  • A disk alert that Tares stores lossless and can correlate with your other sources through the AI SRE workflow.

When something else fits

  • If the monitor’s payload carries nested data you want as labels, flatten it in the Datadog payload first, because event_type_field and text_template read top-level fields only.
  • If your disk alerts come from Prometheus rules routed through Alertmanager, use the alertmanager connector instead. It makes one event per alert and has its own ingest URL.

Still stuck? Get help

Last updated on