What should Datadog monitors about error spikes include for an AI SRE?

A Datadog monitor for an error spike should send Tares four flat fields: a service name, a severity, a summary and a start time. Tares receives them through the webhook connector, and the service name ties the alert to the metrics, logs and deploys already on that service’s timeline. Tares has no connector of its own for Datadog today, so the monitor reaches Tares through a Datadog webhook and a vendor setup.

How it works

Datadog calls a webhook whose URL is the Tares ingest URL. You write the payload yourself in the webhook Payload field, using Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so the shape is yours. Tares stores the payload lossless and maps the fields you declare into labels.

Four fields are worth sending for an error spike:

• service: the name of the failing service. Make it the primary label. The AI SRE guide keys every source by the same service label, so the alert lands on one timeline next to metrics, logs and deploys, and the agent can tie the spike to evidence.
• severity: how urgent the monitor is, so a trigger or the agent can treat a critical spike differently from a warning.
• summary: one line on what happened. Build it from $EVENT_MSG so the agent reads the monitor’s own words.
• start time: when the spike began, as an ISO-8601 timestamp. Tares reads it through event_time_field, so the event sits at the right point on the timeline, beside a deploy that landed minutes earlier.

Keep the payload flat. text_template and event_type_field read top-level fields only, so nested objects are stored but cannot feed the text or the event type. Adding $LINK as a further top-level field gives the agent a way back to the monitor.

Set it up

1. Create a webhook source in Tares under Sources, Add source, webhook, or ask a connected agent to create it. Declare a label per field and mark service as primary. Save, then copy the ingest endpoint from the source page.

- name: ci-events
  connector: webhook
  config:
    event_time_field: started_at
    text_template: '{severity} {summary}'
    labels:
      - { name: service, field: service, primary: true }
      - { name: severity, field: severity }

Change the name to something like a Datadog errors source. The labels, text_template and event_time_field keys work the same way.

2. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL.
3. In the Payload field, write a flat JSON object with service, severity, summary and started_at, filling the values from Datadog variables such as $EVENT_MSG and adding $LINK if you want the monitor link.
4. When Tares has auth on, add custom headers as JSON with an Authorization Bearer header that holds an ingest-scoped API key. On a secured instance this header is the credential.
5. Put @webhook- followed by your webhook’s name in the monitor message so the monitor calls it.
6. Check the mapping by posting a sample payload to the ingest URL, then open Recent events on the source. Fields shows which payload fields arrive and how often.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"service": "checkout", "severity": "critical", "summary": "5xx rate above threshold", "started_at": "2024-01-01T00:00:00Z"}'

What you get

• Each error-spike alert arrives as an event labeled by service, on the same timeline as your other sources for that service.
• The event text reads as severity and summary, so the agent sees what fired without parsing raw JSON.
• The start time places the spike on the timeline, so the agent can compare it with deploys and config changes.
• The full payload is stored lossless, so fields you did not map, such as the monitor link, stay available.
• Triggers can watch the Datadog source by service and wake an agent when an alert arrives.

When something else fits

• If your alerts already come from Prometheus rules, use the prometheus_alerts or alertmanager connectors described in the AI SRE guide. They carry labels and annotations in a fixed shape, so you write no payload.
• If you need to ingest anything else that can send an HTTP request, such as CI events or deploy notices, the same webhook connector covers it with the same source setup.

Related