What should an AI SRE read next to Datadog monitors for error spikes?

Tares puts the Datadog monitor alert on one per-service timeline next to that service’s metrics, logs and recent deploys, so an AI SRE reads all of it together. The diagnosis then cites evidence lines instead of guessing from the alert text. Datadog reaches Tares through the webhook connector and a Datadog-side webhook setup, because Tares has no connector of its own for Datadog today.

How it works

An error spike alert tells you that something broke, not why. Tares keeps the evidence for a diagnosis on one timeline per service:

• The alert. Your Datadog monitor calls a Datadog webhook. The webhook posts JSON to a Tares ingest URL, and the webhook connector stores the payload lossless as an event.
• Metrics. A prometheus source carries the service’s 5xx rate, latency and similar signals.
• Logs. A docker_logs source tails the container’s logs, or an OTLP source carries your OpenTelemetry logs.
• Recent changes. A second webhook source receives a deploy event from your CD pipeline on every deploy, so the timeline shows a deploy landing shortly before the errors started.

Every source carries the same service label, and that shared label is what merges them onto one clock. A trigger on the alert event wakes your agent, or a Tares agent, with that correlated timeline. The agent reads the alert, the metric that moved, the error log lines and the nearest deploy in a single read, then writes its finding back to the same timeline. The next investigation starts from that conclusion. The Build an AI SRE guide walks through this whole loop end to end.

Set it up

1. Create a webhook source for the Datadog alerts. In the console go to Sources, Add source, webhook. Declare a label for each payload field you want to correlate on and mark one as primary. Make the primary label service, so the alert lands on the same timeline as your metrics and logs. Save, then copy the ingest endpoint from the source page. The shape of a webhook source looks like this:

- name: ci-events
  connector: webhook
  config:
    event_type_field: action
    text_template: '{action} by {sender}'
    labels:
      - { name: pipeline, field: pipeline, primary: true }

2. Add the webhook in Datadog. Open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as JSON from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK. Add a field that carries the service name, so the service label has something to read. Keep the payload flat, because event_type_field and text_template read top-level fields only.

3. Authenticate if your instance is secured. When Tares has auth on, add custom headers as JSON in the Datadog webhook, with an Authorization Bearer header that holds an ingest-scoped API key. That header is the credential.

4. Make the monitor call the webhook. Put the webhook’s @webhook- handle, followed by its name, in the monitor message.

5. Feed the change signal. Create a second webhook source with a primary service label, event_type: deploy and a text template like deploy {commit} by {author}: {message}. Then add one line to your CD pipeline, run on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Connect metrics and logs. Point a prometheus source at your metrics and a docker_logs or OTLP source at your logs, each with the same service label.

7. Verify. Trigger a test alert or POST a test payload, then check Recent events on the source. The Fields view shows which payload fields arrive and how often, which helps you pick a reliable key. Then open Explore for the service and confirm the alert, metrics, logs and deploy sit on one timeline.

What you get

• A Datadog error spike alert that arrives with the evidence around it: the service’s metrics, its logs and its latest deploys, merged on one clock.
• A diagnosis that ties the failure to specific evidence lines, such as “deploy landed at 14:02, errors started at 14:04”, instead of restating the alert.
• A finding written back to the timeline, so the next investigation, yours or an agent’s, starts from the previous conclusion.
• A trigger that wakes your agent when the alert event arrives, so nobody has to poll and the diagnosis can be waiting before you look.
• Datadog keeps owning the alerting. Tares ingests what fired and correlates it, and nothing is re-thresholded.

When something else fits

• A Prometheus-based stack. If your alerts come from Prometheus rules or Alertmanager, use the alert connectors that exist for them. They keep alerting where it is, and Tares ingests what fires.
• A different vendor. Tares has no connector of its own for Datadog, Sentry, PagerDuty or other vendors today. The webhook connector covers any of them that can send an HTTP request, with a vendor-side setup like the one above.
• Alerts without a service key. If the Datadog payload cannot carry a service name that matches your other sources, the alert will not join the service’s timeline. Add the service field to the payload first.
• Reading the alert alone. If the alert text is all you need, the webhook source and its Recent events view are enough. Add metrics, logs and deploys when you want the diagnosis to cite evidence.

Related