How do I route Datadog monitors about a cron job that stopped to an AI SRE?

Tares receives Datadog monitors through the webhook connector and a Datadog webhook setup, and a trigger that filters on one symptom wakes your AI SRE only for those alerts. You point a dedicated Datadog webhook at a Tares ingest URL, and the payload names the symptom, here a cron job that stopped. Tares has no connector of its own for Datadog today, so the webhook connector is the route. Monitors about other problems never reach the agent through this trigger.

How it works

• Datadog calls a webhook you define in its Webhooks integration. The URL is the Tares ingest URL of a webhook source, and the JSON payload is written by you from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK.
• Tares stores the payload lossless. The source reads a top-level payload field as the event type and renders the event text from a template, so the symptom is named on every event.
• A trigger watches that source and keeps only events whose type is the symptom you chose. When the condition holds, the trigger wakes your agent. Read Build an AI SRE for how triggers, agents and the correlated timeline fit together.
• Keep the payload flat. event_type_field and text_template read top-level fields only, so a nested payload cannot drive either of them.
• The source is a webhook connector source. Datadog does not need a confirmation handshake, so the first delivery is just a normal event.

Set it up

1. Create a webhook source in Tares under Sources → Add source → webhook, or ask a connected agent to create it. Declare a label for the field you want to key on and mark it primary. Use the config below, which reads the event type from a field called symptom:

- name: datadog-monitors
  connector: webhook
  config:
    event_type_field: symptom
    text_template: '{symptom}: {job}'
    labels:
      - { name: job, field: job, primary: true }

2. Copy the ingest endpoint from the source page. The ingest URL has the form /ingest/ followed by the ingest key.
3. In Datadog, open the Webhooks integration and add a webhook whose URL is that Tares ingest URL. Give it a name that says what it is for, such as a cron-stopped webhook.
4. Write the payload as flat JSON. Include one field that carries a fixed symptom value of cron_stopped and one field that carries the job, filled from a Datadog variable. Name them symptom and job to match the source above.
5. If Tares has auth on, add custom headers as JSON in the Datadog webhook with an Authorization Bearer header holding an ingest-scoped API key. On an instance without auth you need no header.
6. In the message of each Datadog monitor that watches a cron job, put @webhook- followed by your webhook name. Monitors about other problems do not carry it, so they never post to this source.
7. Check the mapping by posting a test event, then open Recent events and Fields on the source page:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"symptom": "cron_stopped", "job": "nightly-backup"}'

8. Add a trigger that keeps only that symptom and wakes your agent. The trigger name states the symptom, and the filter makes it the only thing that fires it:

- name: cron-stopped
  sources: [datadog-monitors]
  filters:
    - { field: event_type, op: eq, value: cron_stopped }
  condition: { aggregate: count, predicate: "> 0", window: 5m }

9. Choose who gets woken for this trigger: a Tares agent, your own agent service over a webhook, or a Slack channel.

What you get

• Only Datadog monitors that carry your webhook handle reach the source, and only events of type cron_stopped fire the trigger, so the agent wakes for the symptom you chose.
• Each event text names the symptom and the job, for example cron_stopped: nightly-backup, so the agent and anyone reading Recent events see what happened without opening the raw payload.
• The full Datadog payload is kept lossless, so the $LINK and $EVENT_MSG content you included stays available on the event.
• One source and one trigger per symptom. A second symptom gets its own Datadog webhook and trigger, and the two do not interfere.

When something else fits

• If you want every Datadog monitor to wake the agent, drop the symptom filter from the trigger and keep the source as it is.
• If the alert is fired by Prometheus rules rather than Datadog, use the alertmanager or prometheus_alerts connector instead of a webhook.
• If your payload must stay nested, flatten it in the Datadog Payload field first, because the event type and text template read top-level fields only.

Related