How do I route Datadog monitors about a disk filling up to an AI SRE?

Tares receives Datadog monitors through the webhook connector, and a trigger with a filter on one symptom wakes your AI SRE only for disk-filling-up alerts. Datadog reaches Tares through the webhook connector and a Datadog-side setup, because Tares has no connector of its own for Datadog today. You write the Datadog payload, so you put the symptom in a field, filter on it in the trigger, and name the trigger after the symptom.

How it works

Datadog posts each monitor notification to a Tares ingest URL. Tares stores the payload lossless as an event on a webhook source. A trigger watches that source, keeps only the events whose symptom field matches, and wakes the agent when the condition holds.

• The payload is yours. In the Datadog Webhooks integration you write the JSON from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK. Add a field you name, for example symptom, with a fixed value such as disk_filling_up.
• Keep it flat. event_type_field and text_template read top-level fields only, so put every field you need at the top level.
• The filter does the narrowing. Other Datadog monitors can share the same source. Only events that pass the trigger’s filter count toward the condition.
• The trigger names the symptom. Call it disk-filling-up. Its firings show up under that name, and the agent subscribed to it gets a prompt written for that symptom. This is the same pattern as the incident trigger in the AI SRE guide, where Datadog stays your alerting and Tares handles the correlation and the waking.

Set it up

1. Start Tares and open the console. Run this in a terminal:

tares up

2. Create a webhook source under Sources, Add source, webhook, or ask a connected agent to do it. Declare a label per field you want to correlate on and mark one primary. Use the fields your Datadog payload carries:

- name: datadog-monitors
  connector: webhook
  config:
    event_type_field: symptom
    text_template: '{symptom}: {message}'
    labels:
      - { name: host, field: host, primary: true }
      - { name: symptom, field: symptom }

3. Copy the ingest endpoint from the source page. Use test_source with a sample payload to check that the labels map before you wire Datadog.

4. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the flat JSON payload with your symptom, message and host fields, using Datadog variables for the values that change. If Tares runs with auth on, add custom headers as JSON with an Authorization Bearer header that holds an ingest-scoped API key. On a secured instance that header is the credential.

5. Put @webhook- followed by your webhook name in the message of each disk monitor, so the monitor calls it.

6. Create the trigger. It filters to the one symptom, counts matching events in a window, and waits out a cooldown so a flapping monitor does not wake the agent repeatedly:

- name: disk-filling-up
  sources: [datadog-monitors]
  filters:
    - { field: symptom, op: eq, value: disk_filling_up }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 30m

7. Subscribe an agent to the trigger. A Tares agent is defined by its prompt, so write one about a filling disk: what to check first and what next action to suggest. To use your own agent service, deliver the trigger to it over a webhook.

8. Verify. Trigger a disk monitor in Datadog, then check Recent events on the source and the trigger’s firings. Fields on the source shows which payload fields arrive and how often.

What you get

• Only disk-filling-up alerts wake the agent. Every other Datadog monitor can use the same source without firing this trigger.
• A trigger whose name states the symptom, so each firing is easy to recognize in the console.
• An agent whose prompt is written for one symptom instead of a generic alert.
• Every Datadog payload stored lossless, so the agent can read fields you did not map into labels.
• Datadog stays your alerting. Tares ingests what fires, correlates it with your other sources and handles the waking.

When something else fits

• If every monitor on a webhook is already about disks, you can skip the symptom filter and let the whole source drive the trigger.
• If you want a separate agent and cooldown for each symptom, add one trigger per symptom on the same source.
• If your alerts come through Alertmanager rather than Datadog, use the alertmanager connector, which makes one event per alert.

Related