How do I route Datadog monitors about high latency to an AI SRE?

Tares routes Datadog high-latency monitors to an AI SRE through a webhook source and a trigger that filters on one symptom. Datadog calls the Tares ingest URL, and only events that carry your high-latency symptom value fire the trigger and wake the agent. Tares has no connector of its own for Datadog today, so the monitors reach Tares through the webhook connector and a Datadog-side webhook setup.

How it works

Three pieces split the work, so only the alerts you care about wake the agent.

• Datadog chooses what is sent. A monitor calls a webhook only when its message contains @webhook-<name>. Put that handle on your latency monitors and leave it off the rest.
• The payload names the symptom. In the Datadog webhook you write the payload as a JSON object, using Datadog variables such as $EVENT_MSG for the monitor text. Add a fixed field of your own that says which symptom this is. The shape is yours, so keep it flat: event_type_field and text_template read top-level fields only.
• The trigger filters on the symptom. A webhook source in Tares stores each payload lossless and maps the fields you declare into labels. A trigger then lists the source, filters on the symptom field, and fires only on matches.

Your agent gets woken by that trigger, so latency monitors reach it and unrelated Datadog alerts do not. You pick who is woken per trigger: a Tares agent, your own agent service over a webhook, or a Slack channel. The AI SRE guide covers that choice.

Set it up

1. Create the webhook source in Tares. Go to Sources → Add source → webhook. Declare a label for each payload field you want to filter or correlate on, and mark one primary. Set text_template so the symptom leads the event text. Save, then copy the ingest endpoint from the source page.

- name: datadog-latency
  connector: webhook
  config:
    text_template: '{symptom}: {message}'
    labels:
      - { name: symptom, field: symptom, primary: true }
      - { name: message, field: message }

2. Add the webhook in Datadog. Open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. In the Payload field, write a flat JSON object from Datadog variables. Include a symptom field with the fixed value high_latency and a message field built from $EVENT_MSG.

3. Add the credential if Tares has auth on. In the webhook’s custom headers, written as JSON, add an Authorization Bearer header that holds an ingest-scoped API key. On a secured instance this header is the credential. Without auth you can skip it.

4. Point the latency monitors at it. Put @webhook-<name> in the message of each high-latency monitor, using the name you gave the webhook.

5. Check the mapping before you rely on it. Post a test payload and look at Recent events. Fields shows which payload fields arrive and how often.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"symptom": "high_latency", "message": "p99 above 1s"}'

6. Create the trigger. Name it for the symptom, list the webhook source, and filter on the symptom field so only high-latency events count. The cooldown keeps one noisy monitor from waking the agent again and again.

- name: high-latency
  sources: [datadog-latency]
  filters:
    - { field: symptom, op: eq, value: high_latency }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 5m

7. Choose who is woken. Subscribe a Tares agent to the high-latency trigger, or send the firing to your own agent service or a Slack channel.

What you get

• Only Datadog monitors that carry your @webhook-<name> handle and your high_latency value can fire the trigger. Other Datadog alerts either never reach the source or fail the filter.
• Each event’s text starts with the symptom, because text_template renders it first. A trigger named high-latency makes the firings easy to find on the trigger’s page and under Firings.
• The full Datadog payload is stored lossless, so fields you did not map are still there to read.
• A cooldown on the trigger limits repeat wake-ups from the same noisy monitor.
• You can repeat the pattern for another symptom, such as errors or saturation. Add a second webhook source or a second trigger with its own symptom value.

When something else fits

• If you want every Datadog alert to reach the agent with no symptom filter, use a single webhook source and a trigger without the symptom filter.
• If your latency alerts come from Prometheus or Alertmanager instead of Datadog, the Alertmanager and Prometheus alerts connectors fit better. Tares ingests what those fire, and you keep alerting where it is.

Related