What should an AI SRE read next to Datadog monitors for failed deploys?

Tares puts a Datadog monitor alert on one timeline with the same service’s metrics, logs and deploys, so an AI SRE reads the evidence next to the alert. For a failed deploy, the agent sees the alert, the errors and the deploy event on one clock, and its diagnosis cites those lines. Datadog reaches Tares through the webhook connector and a Datadog-side setup, because Tares has no connector of its own for Datadog today.

How it works

The alert is only the start of an investigation. Tares keeps one timeline per service, built from sources that share the same service label. Whatever the agent reads for that service comes back merged on one clock.

For a failed deploy, the timeline holds four kinds of evidence:

• The Datadog monitor alert. It arrives through a webhook source. Datadog posts the JSON you write in the webhook Payload field, and Tares stores it lossless.
• The service’s metrics. A prometheus source polls the metrics you pick, such as error rate and latency, with service read from the metric labels.
• The service’s logs. A docker_logs source tails the container’s logs. An otlp source works if you run an OpenTelemetry pipeline.
• Recent changes. A second webhook source receives one event per deploy from your CD pipeline. A deploy that lands at 14:02 sits next to errors that start at 14:04.

The agent reads that timeline over MCP, or a trigger keyed by service wakes it when the alert event arrives. Because the evidence is on the same timeline, the agent ties its conclusion to specific lines: the monitor that fired, the error logs, the metric that moved and the deploy before it. It writes the finding back onto the timeline, so the next investigation starts from it. The AI SRE guide walks through this loop end to end.

Set it up

1. Create a webhook source for Datadog. In Sources → Add source → webhook, declare a label for each payload field you want to correlate on, and mark one primary. Make service the primary label so the alert lands on the same timeline as your other sources. Copy the ingest endpoint from the source page.

- name: datadog-monitors
  connector: webhook
  config:
    labels:
      - { name: service, field: service, primary: true }

2. Add the webhook in Datadog. Open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as JSON from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, and include the service name as a top-level field. Keep the payload flat, because event_type_field and text_template read top-level fields only.
3. Authenticate if Tares has auth on. Add custom headers as JSON with an Authorization Bearer header that holds an ingest-scoped API key. On a secured instance that header is the credential.
4. Call the webhook from the monitor. Put @webhook-<name> in the monitor message, using the name you gave the webhook in Datadog.
5. Add a deploy source. Create a second webhook source with a primary service label, event_type: deploy and a text template over your payload fields.

- name: deploys
  connector: webhook
  config:
    event_type: deploy
    text_template: 'deploy {commit} by {author}: {message}'
    labels:
      - { name: service, field: service, primary: true }

6. Post a deploy event from your pipeline. Run one line in your CD pipeline on every deploy.

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

7. Key metrics and logs by the same label. Point a prometheus source at your metrics and a docker_logs or otlp source at your logs, each with a service label. Then open Explore, pick the service, and check that the alert, metrics, logs and deploy appear together. Recent events and Fields on each webhook source show what arrives.

What you get

• One timeline per service with the Datadog alert, metrics, logs and deploys on a single clock.
• A diagnosis that cites evidence: the monitor that fired, the log lines, the metric and the deploy that came just before.
• A deploy source that gives the agent the most common root cause, a change, instead of leaving it to guess.
• A finding written back to the timeline, so the next investigation starts from the previous conclusion.
• Datadog keeps owning alerting. Tares ingests what fired, correlates it with the evidence and handles the waking.

When something else fits

• If your alerts already come from Prometheus rules, use the prometheus_alerts or alertmanager connector instead. Those have their own Tares connectors, so no Datadog webhook is needed.
• If you only need to look at Datadog alerts by themselves, the webhook source alone stores them, but the agent then has no metrics, logs or deploys to cite.

Related