How do I wake an AI SRE once when Datadog monitors report error spikes?

Tares wakes your AI SRE once per entity when Datadog monitors report error spikes. Datadog posts to a Tares webhook source, a trigger watches it, and the trigger’s cooldown sets the minimum gap between firings for the same entity. Tares has no connector of its own for Datadog today, so the alerts arrive through the webhook connector and a Datadog webhook setup.

How it works

• Datadog posts to a webhook source. The webhook connector accepts JSON at the source’s ingest URL and stores the payload lossless. Labels map payload fields into the event.
• A trigger watches the source. A trigger evaluates its condition as events arrive. With group_by: [key_value] it evaluates per entity, so each service fires independently.
• The cooldown limits repeat firings. cooldown is the minimum gap between firings for the same entity. It prevents a flapping condition from firing repeatedly, so a monitor that keeps re-alerting wakes your agent once per cooldown window for each service.
• The dispatch carries context. When the condition holds, Tares POSTs one JSON body to each subscriber. Its payload is the entity’s correlated timeline across the trigger’s sources over context_window, so the agent starts with context and makes no follow-up call.
• Every firing is logged. You can see firings and per-subscriber delivery status in the console under Firings, on the trigger’s page, and on the project’s Activity.

Set it up

1. Create the webhook source. In the console, go to Sources → Add source → webhook, or ask a connected agent to create it. Declare a label for the field that names the entity and mark it primary. Declare a numeric label for the field you aggregate, because triggers aggregate a typed field. Save, then copy the ingest endpoint from the source page.

- name: datadog-monitors
  connector: webhook
  config:
    labels:
      - { name: service, field: service, primary: true }
      - { name: value, field: value, type: number }

2. Add the webhook in Datadog. Open the Webhooks integration and add a webhook whose URL is the Tares ingest URL.
3. Write a flat payload. In the webhook’s Payload field, write JSON from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, and include the fields your labels read. Keep it flat, because event_type_field and text_template read top-level fields only.
4. Add the credential if Tares has auth on. Add custom headers as JSON with an Authorization Bearer header holding an ingest-scoped API key. On a secured instance this header is the credential.
5. Call the webhook from the monitor. Put @webhook- followed by your webhook’s name in the monitor message.
6. Test the mapping. Post a sample payload, then check Recent events and Fields on the source page.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "deploy", "sender": "ci", "pipeline": "web"}'

7. Define the trigger with a cooldown. Add it on the project’s page, under Triggers → Add trigger, or in the catalog YAML. Adjust the predicate and window to your error spike.

triggers:
  - name: error_spike
    sources: [datadog-monitors]
    key_field: service
    condition:
      aggregate: max
      field: value
      predicate: "> 0"
      window: 1m
      group_by: [key_value]
    emit:
      kind: error_spike
      context_window: 15m
    cooldown: 5m

8. Subscribe your agent. A Tares agent is woken by the trigger in a project that wires it to that trigger. An external agent registers a webhook with the subscribe tool and acknowledges each dispatch with a 2xx.

What you get

• One investigation per entity per cooldown window, instead of one per repeated Datadog notification.
• A dispatch whose payload is the entity’s correlated timeline over context_window, ready to hand to the model.
• Independent firing per service, because the condition is evaluated per entity.
• A record of every firing and its delivery outcome, even with zero subscribers.
• Retries for delivery failures. Tares retries 5xx responses and transport errors with backoff. Delivery is at-least-once, so dedupe on dispatch_id.

When something else fits

• You want a check on a clock, not on a condition. Give the trigger a schedule with condition: {every: 10m}. It fires once per interval and hands over a summary of the window. See Watch with triggers.
• Your payload has no numeric field. A condition aggregates a typed field, so add a numeric label to the payload. You can also add a second source that produces one, and have the trigger watch both.
• A 4xx from your endpoint. Tares records it as a failed delivery and does not retry, so fix the endpoint rather than waiting on retries.
• You need a different shape of incident. Pick different aggregates and fields per trigger, for example one trigger for latency and another for an error storm.

Related