How do I route Datadog monitors about crash loops to an AI SRE?

Tares routes Datadog crash loop monitors to an AI SRE through a webhook source and a trigger that filters on one symptom. Datadog posts to the Tares ingest URL, and only alerts that carry the crash loop symptom wake the agent. Tares has no connector of its own for Datadog today, so Datadog reaches Tares through the webhook connector and a Datadog-side setup.

How it works

Datadog reaches Tares through the webhook connector and a vendor setup in Datadog. Your Datadog webhook posts JSON to the source’s ingest URL, and Tares stores the payload lossless.

You write the payload yourself in the Datadog webhook Payload field, using Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK. Because the shape is yours, you decide which fields identify the service and the symptom. Keep the payload flat: event_type_field and text_template read top-level fields only.

Three parts do the routing:

• A webhook source with labels for the fields your payload carries. One label is primary, and a service label puts each alert on that service’s timeline next to your metrics and logs.
• A trigger with a filter on the symptom field. Alerts without the crash loop symptom never match, so they never wake the agent.
• An agent subscribed to the trigger. The trigger is named for the symptom, so each firing is recorded under a crash loop name, and the agent’s prompt tells it to treat the alert as a crash loop.

The AI SRE guide shows the same pattern end to end: sources keyed by service, a trigger that fires on an alert event, and an agent that reads the correlated timeline and writes a finding back.

Set it up

1. Create a webhook source for the Datadog monitors. In the console go to Sources → Add source → webhook, or ask a connected agent to create it. Mark one label primary and add a label for the symptom field.

- name: datadog-monitors
  connector: webhook
  config:
    text_template: '{symptom} on {service}: {summary}'
    labels:
      - { name: service, field: service, primary: true }
      - { name: symptom, field: symptom }

2. Copy the ingest endpoint from the source page. In Datadog, open the Webhooks integration and add a webhook whose URL is that Tares ingest URL.

3. Write the payload as flat JSON with three fields: service, symptom and summary. Fill service and summary from Datadog variables such as $EVENT_MSG, and set symptom to a fixed value for this webhook, for example crash_loop. Name the webhook for the symptom so it is clear what it carries.

4. If Tares runs with auth on, add custom headers as JSON in the Datadog webhook with an Authorization Bearer header holding an ingest-scoped API key. Without auth, skip this.

5. Add @webhook- followed by your webhook’s name to the message of each crash loop monitor, so those monitors call it. Monitors for other symptoms do not use this webhook.

6. Send a test event and check Recent events on the source page. Fields shows which payload fields arrive.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"service": "checkout", "symptom": "crash_loop", "summary": "checkout pods restarting"}'

7. Create the trigger. Its filter keeps only crash loop alerts, and its name carries the symptom.

- name: datadog-crash-loop
  sources: [datadog-monitors]
  filters:
    - { field: symptom, op: eq, value: crash_loop }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 10m

8. Subscribe an agent to the trigger. For a Tares agent, write a prompt that says the alert is a crash loop: find the restarting service, tie the cause to evidence lines on its timeline, and suggest a next action. To wake your own agent service instead, deliver the trigger over a webhook.

What you get

• Only the alerts you tag as crash loops wake the agent. Other Datadog monitors can still reach Tares, but they do not match this trigger.
• Each firing shows under the trigger named datadog-crash-loop, so you see which symptom woke the agent without opening the alert.
• The agent reads one correlated timeline for the service: the Datadog alert, plus any metrics, logs and deploys you have wired in with the same service label.
• The agent writes its finding back to the timeline, so the next investigation starts from the previous conclusion.
• The cooldown keeps a monitor that keeps re-alerting from waking the agent again and again.
• Adding another symptom later means one more webhook payload value and one more trigger.

When something else fits

• If your alerts already flow through Prometheus rules, use the prometheus_alerts or alertmanager connectors and keep alerting where it is.
• If you want every Datadog alert to reach the agent regardless of symptom, drop the filter and trigger on the source alone.
• If one webhook must carry several symptoms, put the symptom in the payload per monitor and use a filter with op: in listing the values you want.

Related