What should an AI SRE read next to Datadog monitors for a cron job that stopped?

Tares lets an AI SRE read the Datadog alert together with the same service’s metrics, logs and recent deploys or config changes on one timeline. Datadog reaches Tares through the webhook connector and a Datadog-side setup, because Tares has no connector of its own for Datadog today. The agent then cites evidence lines in its diagnosis instead of reasoning from the alert alone.

How it works

The Datadog monitor is the start of the investigation, not the evidence. Tares merges every source that carries the same service label onto one timeline per service, so the alert sits on one clock beside everything else about that service.

For a cron job that stopped, read these next to the monitor alert:

• Metrics for the same service, from a prometheus source. They show whether the job’s numbers flatlined and when.
• Logs for the same service, from a docker_logs source or from OTLP. They show the job’s last output before it went quiet.
• Recent changes for the same service, from a webhook source that your CD pipeline posts deploy and config events to. Most real root causes are a change, so a deploy landing shortly before the job stopped is the evidence you want next to the alert.

Datadog reaches Tares as a webhook source. You write the payload in Datadog’s webhook Payload field as JSON, built from Datadog variables such as $EVENT_MSG, so the shape is yours. Keep it flat, because event_type_field and text_template read top-level fields only. Add a service field so the alert joins the same timeline as the metrics, logs and changes.

A trigger can wake an agent when the alert event arrives. The agent reads the correlated timeline, ties its conclusion to specific evidence lines and writes a finding back, so the next investigation starts from it. The Build an AI SRE guide walks through this loop end to end.

Set it up

1. Create the Datadog source. In the console, go to Sources, Add source, webhook. Declare a label for each payload field you want to correlate on, and mark one primary. Use service as the primary label, read from the payload’s service field, so the alert lands on the service’s timeline. Save, then copy the ingest endpoint from the source page.

   - name: datadog-monitors
     connector: webhook
     config:
       labels:
         - { name: service, field: service, primary: true }

2. Add the webhook in Datadog. Open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as flat JSON from Datadog variables, including the service field.

3. Authenticate if Tares has auth on. Add custom headers as JSON with an Authorization Bearer header holding an ingest-scoped API key. On an instance without auth you can leave this out.

4. Make the monitor call it. Put @webhook-<name> in the monitor message, using the name you gave the webhook.

5. Add the evidence beside it. Keep your metrics and log sources keyed by the same service label. Then create a second webhook source for changes, with a primary service label and event_type: deploy. Run one line in your CD pipeline on every deploy:

   curl -X POST https://<your-tares>/ingest/<ingest_key> \
     -H 'content-type: application/json' \
     -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Check the mapping. POST a test payload and look at Recent events on the source page. Fields shows which payload fields arrive and how often, which tells you whether service is reliable. The webhook connector page covers the options.

7. Wake an agent. Create a trigger over these sources, keyed by service, and subscribe a Tares agent, your own agent service or a Slack channel to it.

What you get

• One timeline per service with the Datadog alert, the metrics, the logs and the recent deploys or config changes merged on one clock.
• A diagnosis that cites evidence lines, such as a deploy that landed before the job stopped, instead of restating the alert.
• A finding written back onto the service’s timeline, so the next investigation, yours or an agent’s, starts from the earlier conclusion.
• Datadog keeps owning the monitor. Tares ingests what fired, correlates it and handles the waking, and nothing is re-thresholded.

When something else fits

• If you already run Prometheus rules or Alertmanager, their own connectors take the alerts directly, so Datadog is only needed if Datadog owns your monitors.
• If the job’s logs or metrics are not in any source Tares reads, the timeline holds the alert and the changes only, so connect those signals first.

Related