How do I route Datadog monitors about failed deploys to an AI SRE?

Tares receives Datadog monitors through the webhook connector and a Datadog webhook setup, and a trigger filtered to one symptom wakes your AI SRE only for failed deploys. Tares has no connector of its own for Datadog today. You write the Datadog payload yourself, so you can put the symptom in it and filter on it. The agent then starts from the correlated timeline, with the symptom in the event text.

How it works

• Datadog calls a webhook you define in its Webhooks integration. The URL is a Tares ingest URL, and the payload is a JSON object you write from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK.
• A Tares webhook source stores the payload lossless and maps the fields you declare as labels. Keep the payload flat, because event_type_field and text_template read top-level fields only.
• You choose one top-level field in the payload to carry the symptom, for example a field with the fixed value failed-deploys. The source reads it as the event type with event_type_field, and text_template can put it in the event text.
• A trigger lists that source, filters on the symptom field, and applies a condition and a cooldown. Only matching events can fire it, so other Datadog alerts do not wake the agent.
• When the trigger fires, it wakes the agent you subscribed to it. Each firing is listed on the trigger’s page and under Firings in the console.

In the Build an AI SRE guide, the agent reads one correlated timeline per service, so the failed deploy appears next to the metrics, logs and alerts for the same service.

Set it up

1. Create a webhook source in the console (Sources → Add source → webhook) or ask a connected agent to create it. Declare a label for each payload field you want to correlate on and mark one primary. Copy the ingest endpoint from the source page. The webhook connector page covers the source options.

- name: datadog-deploys
  connector: webhook
  config:
    event_type_field: symptom
    text_template: '{symptom}: {monitor} on {service}'
    labels:
      - { name: service, field: service, primary: true }
      - { name: monitor, field: monitor }

2. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write a flat JSON payload that includes the symptom, the service, the monitor name and the Datadog variables you want, such as $EVENT_MSG and $LINK. The payload fields must match the labels and the template in step 1.
3. If Tares has auth on, add custom headers as JSON with an Authorization Bearer header that holds an ingest-scoped API key. Without auth on, you do not need the header.
4. Put @webhook- followed by your webhook name in the message of each monitor that watches failed deploys. Monitors without that mention never call it, which is the first filter.
5. Add a trigger over the source that matches only the symptom. The trigger shape is the one from the webhook connector and AI SRE guides:

- name: failed-deploys
  sources: [datadog-deploys]
  filters:
    - { field: event_type, op: eq, value: failed-deploys }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 5m

6. Subscribe a Tares agent to the trigger, or deliver it to your own agent over a webhook or to a Slack channel. Edit the agent prompt to say what a failed-deploy investigation should check.
7. Fire a test monitor. Check Recent events on the source and Fields to confirm the payload fields arrive, then check the trigger’s page for the firing.

What you get

• Only Datadog alerts that carry the failed-deploys symptom can fire the trigger. Other monitors can post to the same source without waking the agent.
• The symptom is in the event text through text_template, so what lands on the timeline reads as a failed deploy.
• The agent investigates from one correlated timeline per service, so it can compare the failed deploy with errors and latency on the same clock.
• Every firing is listed on the trigger’s page and under Firings, so you can see how often the symptom woke the agent.
• The cooldown on the trigger stops repeated alerts for the same symptom from waking the agent again right away.

When something else fits

• If you want the agent woken for every Datadog alert, skip the symptom filter and use a broader condition on the same source.
• If your deploys are visible from your pipeline, a separate webhook source posted from CD gives the agent the deploy itself, as the Build an AI SRE guide shows. Use it next to the Datadog source rather than instead of it.
• If Tares already receives alerts from Prometheus or Alertmanager, filter those sources by label in the same way instead of adding Datadog.

Related