How do I route Datadog monitors about 5xx errors on an API to an AI SRE?

Tares receives Datadog monitors through the webhook connector and a Datadog webhook setup, and a trigger on that source wakes your AI SRE only for the 5xx monitors you route there. Tares has no connector of its own for Datadog today. You add the webhook handle only to the monitors that watch 5xx errors on the API, and the trigger watches only that source.

How it works

The narrowing happens in two places, and each does a different job.

• Datadog chooses what is sent. A monitor calls a webhook only when its message contains that webhook’s handle, which starts with @webhook- followed by the webhook name. Put the handle in the 5xx monitors on your API and leave it out of every other monitor. Those other alerts never reach Tares through this source.
• The Tares source receives them. A webhook source takes each POST to its own ingest URL and stores the payload lossless. You declare labels that map payload fields into the envelope, with one marked primary.
• The Tares trigger wakes the agent. A trigger lists this one source, filters on the event type, and fires when at least one matching event arrives in the window. Only a firing wakes the agent you picked for the trigger.

The payload is a JSON object you write in the Datadog webhook Payload field, built from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK. Keep it flat. The webhook source reads event_type_field and text_template from top-level fields only.

The trigger sits inside the wider AI SRE setup, where Tares correlates evidence per service and your agent does the investigating. The AI SRE guide walks through that whole loop.

Set it up

1. Create the webhook source in Tares. Use Sources → Add source → webhook, or ask a connected agent to create it. Include a flat service field in your Datadog payload and map it to a primary label. Set a fixed event_type so the trigger can name the symptom. Save, then copy the ingest endpoint from the source page.

- name: datadog-api-5xx
  connector: webhook
  config:
    event_type: datadog_5xx
    labels:
      - { name: service, field: service, primary: true }

2. Add the webhook in Datadog. Open the Webhooks integration and add a webhook whose URL is the Tares ingest URL from step 1. Write the payload as JSON from Datadog variables, with the flat service field included.

3. Add the credential if Tares has auth on. In the webhook’s custom headers, written as JSON, add an Authorization header with the value Bearer followed by an ingest-scoped API key. Creating a push source in the console mints one for it and shows it once. If Tares runs without auth, skip this step.

4. Route only the 5xx monitors. Put the webhook handle (@webhook- followed by the webhook name) in the message of each monitor that watches 5xx errors on the API. Leave it out of all other monitors.

5. Create the trigger. Name it for the symptom, so its page and its firings read as 5xx on the API. It watches only the source from step 1.

- name: api-5xx
  sources: [datadog-api-5xx]
  filters:
    - { field: event_type, op: eq, value: datadog_5xx }
  condition: { aggregate: count, predicate: "> 0", window: 5m }
  cooldown: 5m

6. Pick who gets woken. For this trigger choose a Tares agent, your own agent service over a webhook, or a Slack channel, as described in the AI SRE guide.

7. Verify. Let a 5xx monitor fire or send a test from Datadog. Check Recent events on the source, and use Fields to confirm which payload fields arrive and how often. Then check the trigger’s page for the firing.

What you get

• Only the Datadog monitors that carry the handle reach this source, so the agent is not woken by unrelated alerts.
• Each firing is recorded under the trigger’s name, api-5xx, on the trigger’s page and under Firings, so you see at a glance that the wake-up was about 5xx errors on the API.
• The cooldown of 5m keeps a burst of related alerts from waking the agent again and again.
• Every event keeps its full payload, so the agent reads exactly what Datadog sent, next to the service’s other evidence on one timeline.
• The agent you chose investigates and can write its conclusion back where the next investigation finds it.

When something else fits

• If you want a different symptom, such as latency, to wake an agent, give it its own webhook source and its own trigger, each with its own ingest URL. Keep it separate from this one.
• If your alerting runs in Prometheus or Alertmanager, use their connectors from the AI SRE guide instead of going through Datadog.
• If you want every Datadog alert in one place for correlation, add the handle to all monitors and filter in the trigger by a field you write into the payload.

Related