How do I wake an AI SRE once when Datadog monitors report a cron job that stopped?

Tares wakes your AI SRE once per cron job when you send Datadog monitor alerts to a webhook source and put a cooldown on the trigger that watches it. The cooldown is the minimum gap between firings for the same entity, so a monitor that keeps re-alerting starts one investigation instead of many. Every event the webhook receives is stored, so later alerts and recoveries stay on the entity’s timeline.

How it works

Datadog reaches Tares through the webhook connector and a Datadog-side setup. Tares has no connector of its own for Datadog today.

• Datadog posts to a webhook source. The Datadog Webhooks integration sends a JSON payload that you write from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK. The webhook connector stores the payload lossless and maps the fields you declare as labels into the event.
• A trigger evaluates events as they arrive. A trigger watches the source and checks its condition on each ingested event. With group_by: [key_value] it evaluates per entity, so each cron job fires on its own.
• The cooldown holds repeats back. cooldown is the minimum gap between firings for the same entity. While it runs, repeated alerts for the same job do not start another investigation. A different job has its own cooldown and fires independently.
• The firing hands over the timeline. The dispatch carries the entity’s correlated timeline over context_window, so the agent has context without a follow-up call.

For more on the overall flow, see AI SRE.

Set it up

1. Create a webhook source. Add it under Sources → Add source → webhook, or ask a connected agent to create it. Mark one label primary and make it the field that names the cron job. Save, then copy the ingest endpoint from the source page.

- name: datadog-cron
  connector: webhook
  config:
    text_template: '{message}'
    labels:
      - { name: monitor, field: monitor, primary: true }

2. Add the webhook in Datadog. Open the Webhooks integration and add a webhook whose URL is the Tares ingest URL.
3. Write a flat payload. In the Payload field, write a JSON object with a monitor field set to $ALERT_ID and a message field set to $EVENT_MSG. Add $LINK as a third field if you want the monitor link on the event. Keep it flat, because event_type_field and text_template read top-level fields only. Use one Datadog monitor per cron job, so the monitor ID names the job.
4. Add the credential if Tares has auth on. Add custom headers as JSON with an Authorization Bearer header holding an ingest-scoped API key. Without auth on, no header is needed.
5. Call the webhook from the monitor. Put @webhook-<name> in the monitor message, using the name you gave the webhook in Datadog.
6. Define the trigger. Add it under Triggers → Add trigger, or in the catalog YAML. Set cooldown longer than the interval at which the monitor re-alerts.

triggers:
  - name: cron_stopped
    sources: [datadog-cron]
    condition:
      aggregate: count
      predicate: "> 0"
      window: 5m
      group_by: [key_value]
    emit:
      kind: cron_stopped
      context_window: 1h
    cooldown: 1h

7. Choose who wakes. A Tares agent is woken by a trigger in a project that wires it to that trigger, once you turn it on there. An external agent registers a webhook with the subscribe tool, which takes the trigger name and a URL.
8. Check the result. Post a test alert from Datadog and look at Recent events on the source. Fields shows which payload fields arrive, so you can confirm the monitor field is reliable. Firings and per-subscriber delivery status appear under Firings on the trigger’s page.

What you get

• One investigation per stopped job. A monitor that repeats its alert inside the cooldown does not start more investigations for the same job.
• Independent jobs. Because the condition is evaluated per entity, two cron jobs that stop at the same time each wake the agent once.
• A full timeline. Every event the webhook receives is stored on the source. When the monitor sends a recovery, it is stored like any other event and appears on that job’s timeline, alongside the logs and metrics from the other sources on the trigger.
• A logged record. Every firing is logged, even with zero subscribers, and each dispatch has its own page with the delivery outcome and the exact payload that was sent.

The condition counts every event the source receives, so a recovery that arrives after the cooldown has passed counts as an event too. Filters narrow which events count and which are handed over. Leave them out when you want recoveries in the timeline, and size the cooldown to cover how long you want the first alert to stand in for the rest.

When something else fits

• If you want the agent to look at a source on a clock rather than on an alert, give the trigger a schedule with condition: {every: 10m}, as described in the triggers guide.
• If a monitor sends through a different path, such as Alertmanager, use the connector for that path. The webhook source is the right choice for Datadog.
• Delivery to a subscriber is at-least-once, so an external agent should dedupe on dispatch_id.

Related