What should Datadog monitors about failed deploys include for an AI SRE?

A Datadog monitor for failed deploys should send a flat JSON payload with a service name, a severity, a summary and a start time, so Tares can tie the alert to evidence on that service’s timeline. Datadog reaches Tares through the webhook connector and a Datadog-side setup, because Tares has no connector of its own for Datadog today. You write the payload in the Datadog webhook, and Tares maps each field with a label.

How it works

Datadog calls a webhook that posts to a Tares ingest URL. The webhook connector stores the payload lossless and maps the fields you declare as labels into the event envelope. The labels are what put the alert on the right timeline in the AI SRE setup, next to metrics, logs and deploy events for the same service.

Four things are worth putting in the payload:

• Service name. The label that correlates the alert with everything else about that service. Mark it primary. Use the same value your other sources use, or the alert lands on a different timeline from the evidence. A fixed string per monitor works if the monitor covers one service.
• Severity. Lets the agent and your triggers tell a page-worthy failed deploy from a warning.
• Summary. A short sentence on what failed. In Datadog the monitor message variable $EVENT_MSG is the natural source. Tares can render it as the event text with text_template.
• Start time. The moment the problem began. Point event_time_field at a payload field holding an ISO-8601 timestamp, so the alert sits at the right place on the timeline. The agent can then compare it with the deploy and the first errors.

A link back to the monitor ($LINK) and the alert id ($ALERT_ID) are good extras. They give a human a way back to the Datadog view, and the payload is stored in full either way.

Keep the payload flat. event_type_field and text_template read top-level fields only, so nested objects are harder to use in them. You choose the field names, since the Datadog Payload field is a JSON object you write yourself.

Set it up

1. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL.
2. In the webhook’s Payload field, write a flat JSON object with your service, severity, summary and start time fields, plus $LINK or $ALERT_ID if you want them.
3. If Tares has auth on, add custom headers as JSON with an Authorization Bearer header holding an ingest-scoped API key. On an instance without auth, skip this.
4. Put @webhook- followed by your webhook’s name in the monitor message, so the failed-deploy monitor calls it.
5. In Tares, create a webhook source (Sources → Add source → webhook) with a label for each field you send, one of them primary, and copy the ingest endpoint into the Datadog webhook URL. A source for these alerts looks like this:

- name: datadog-failed-deploys
  connector: webhook
  config:
    event_type: deploy_failed
    event_time_field: started_at
    text_template: '{summary}'
    labels:
      - { name: service, field: service, primary: true }

Add a label for severity the same way, one line per field you want to correlate on. Match started_at, summary and service to the names in your Datadog payload.

6. Check the mapping by posting a test payload to the ingest URL, then open Recent events and Fields on the source page to see which fields arrive and how often:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "deploy", "sender": "ci", "pipeline": "web"}'

Replace the body with your own Datadog payload fields when you test it. You can also ask a connected agent to create the source and run test_source with a sample payload first.

What you get

• Each failed-deploy alert lands as one event on the service’s timeline, with the summary as its text and the start time as its position.
• The agent reads the alert together with the metrics, logs and deploy events for the same service, so it can tie the failure to specific evidence instead of working from the alert text alone.
• The full Datadog payload is kept, so extra fields such as the monitor link stay available even if you never made them labels.
• A trigger can wake an agent on these events, and you can filter on the severity or service labels you declared.

When something else fits

• If your alerts already flow through Prometheus rules, the alertmanager or prometheus_alerts connectors keep that path as it is, with no payload to design.
• If a deploy pipeline can post its own deploy events, a separate webhook source for deploys gives the agent the change itself, not only Datadog’s view of the failure.

Related