Skip to Content

How do I route Datadog monitors about a disk filling up to an AI SRE?

Tares receives Datadog monitors through the webhook connector and a Datadog webhook setup, and a trigger filtered to the disk monitors wakes your agent only for that symptom. Tares has no connector of its own for Datadog today, so you point a Datadog webhook at a Tares ingest URL. Other monitors can arrive on the same source, but only the disk ones fire the trigger. The trigger is named for the symptom, so each firing shows which problem woke the agent.

How it works

  • Datadog posts a JSON body you write yourself in the webhook Payload field. The monitor calls the webhook when its message contains @webhook- followed by the webhook’s name.
  • A webhook source in Tares stores the payload lossless and maps the fields you declare as labels into the event, with one label marked primary.
  • Keep the payload flat. event_type_field and text_template read top-level fields only.
  • A trigger watches the source, filters on a field, and fires only when a matching event arrives. Alerts about CPU, latency or anything else land on the timeline but do not fire this trigger.
  • The agent subscribed to the trigger wakes on each firing. Firings are listed on the trigger’s page and under Firings, under the trigger’s name. This is the push half of the flow in Build an AI SRE.

Set it up

  1. Create a webhook source in Tares under Sources → Add source → webhook, or ask a connected agent to create one. Declare a label for each payload field you want to correlate on, and mark one primary. Copy the ingest endpoint from the source page.
- name: datadog-alerts connector: webhook config: text_template: '{monitor}: {message}' labels: - { name: monitor, field: monitor, primary: true } - { name: host, field: host }
  1. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as JSON from Datadog variables. Use flat top-level fields that match your labels, such as the monitor name, the host and the message.
  2. If Tares has auth on (tares up --auth), add custom headers to the Datadog webhook as JSON with an Authorization Bearer header holding an ingest-scoped API key. On an instance without auth the header is not needed.
  3. Put @webhook- followed by your webhook name in the message of each disk monitor, so those monitors call it.
  4. Check the mapping by posting a sample payload to the source ingest URL, then read Recent events and Fields on the source page. Fields shows which payload fields arrive and how often.
curl -X POST http://localhost:8787/ingest/<ingest_key> \ -H 'Content-Type: application/json' \ -d '{"monitor": "Disk space low", "host": "db-1", "message": "disk is 91 percent full"}'
  1. Add a trigger that watches the source and keeps only the disk monitors. Name it for the symptom. The filter lists the names of your disk monitors, so only those events count.
- name: disk-filling-up sources: [datadog-alerts] filters: - { field: monitor, op: in, value: [Disk space low, Disk forecast full] } condition: { aggregate: count, predicate: "> 0", window: 5m } cooldown: 30m
  1. Subscribe your agent to the disk-filling-up trigger. It now wakes only when a disk monitor alert arrives. The cooldown stops a flapping monitor from waking it again for 30 minutes.

What you get

  • Only disk monitor alerts wake the agent. Other Datadog alerts stay on the timeline as context.
  • Each firing is recorded under the name disk-filling-up, so you see the symptom without opening the payload.
  • The full Datadog payload is stored lossless, and the host and monitor labels let the agent line the alert up with other evidence for the same entity.
  • A cooldown keeps repeated alerts from the same monitor from waking the agent again and again.

When something else fits

  • If you want Datadog to decide which monitors notify, use one Datadog webhook per symptom with its own Tares source. Each source then needs no filter, but you manage more sources.
  • If your disk alerts already come from Prometheus rules, the alerts connectors are a closer fit than a vendor webhook. They need no payload you write by hand.

Still stuck? Get help

Last updated on