What should Datadog monitors about a cron job that stopped include for an AI SRE?

Tares gets the most from a Datadog monitor about a stopped cron job when the webhook payload carries four flat fields: a service name, a severity, a summary and a start time. Datadog posts to a Tares webhook source, and the service name ties the alert to the metrics, logs and deploys already on that service’s timeline. Include the Datadog alert link and alert ID too, so the agent can trace the alert back to Datadog.

How it works

Tares has no connector of its own for Datadog today. Datadog reaches Tares through the webhook connector and a Datadog-side setup. You write the payload yourself in the Datadog webhook Payload field, as a JSON object built from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so the shape is yours.

For a stopped cron job, these fields give an agent the most to work with:

• Service name. Use the same value your other sources use for their service label. Tares builds one timeline per service, so a matching value puts the alert next to that service’s metrics, logs and deploys. This is the primary label.
• Severity. It tells the agent how urgent the miss is and lets a trigger filter on it.
• Summary. One line on what stopped, for example the job missed its expected run. $EVENT_MSG is a natural source for it.
• Start time. An ISO-8601 timestamp of when the monitor began alerting. The agent compares it with deploy and error events to see what changed first.
• Alert link and ID. $LINK and $ALERT_ID point back to the monitor for follow-up.

Keep the payload flat. The webhook source reads event_type_field and text_template from top-level fields only, so nested objects are harder to use in the summary text. The full payload is still stored lossless, so extra fields stay available to the agent.

Set it up

1. Create a webhook source in Tares with Sources → Add source → webhook, or ask a connected agent to create it. Declare a label for each field you want to correlate on and mark service as primary. Map the start time with event_time_field. Copy the ingest URL from the source page.

- name: datadog-cron
  connector: webhook
  config:
    text_template: '{service} {severity}: {summary}'
    event_time_field: started_at
    labels:
      - { name: service, field: service, primary: true }
      - { name: severity, field: severity }

2. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL.
3. Write the Payload field as flat JSON. Use one top-level field each for the service, severity, summary and start time, plus $LINK and $ALERT_ID. The start time field must hold an ISO-8601 timestamp, because that is what event_time_field reads.
4. If Tares has auth on, add custom headers to the Datadog webhook as JSON with an Authorization Bearer header holding an ingest-scoped API key. Without auth on, you do not need the header.
5. Put @webhook- followed by your webhook’s name in the monitor message so the monitor calls it.
6. Check the mapping before you rely on it. Post a sample payload with the same field names to the ingest URL, then open the source page and read Recent events and Fields. Fields shows which payload fields arrive and how often.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"service": "nightly-report", "severity": "critical", "summary": "nightly-report cron has not run", "started_at": "2024-01-01T02:00:00Z"}'

7. Trigger the monitor once, for example on a test job, and confirm the real Datadog event shows the same fields in Recent events.

What you get

• Each stopped-cron alert lands as one event on the service’s timeline, with a readable line built from your text_template.
• The event carries the Datadog start time, so the order of events on the timeline matches what happened.
• A trigger can filter on the severity label and wake an agent. The agent reads one correlated timeline instead of one alert in isolation. The AI SRE guide shows how to wire the trigger and the agent.
• The summary, link and alert ID let the agent tie its finding back to the monitor that fired.
• The full payload is kept as sent, so you can add fields later without losing earlier events.

When something else fits

• If the cron job itself can send an HTTP request, it can post to the same webhook source when it finishes or fails, which gives the agent direct evidence next to the Datadog alert.
• If your alerts already flow through Alertmanager or Prometheus rules, use the alertmanager or prometheus_alerts connector instead of a Datadog webhook.

Related