What should an AI SRE read next to Datadog monitors for crash loops?

An AI SRE on Tares reads the Datadog monitor alert beside the same service’s metrics, logs and recent deploys on one timeline, so its crash loop diagnosis cites evidence. Datadog reaches Tares through the webhook connector and a Datadog webhook setup, and Tares has no connector of its own for Datadog today. Every source carries the same service label, so one read returns the alert and everything around it.

How it works

A monitor firing tells you that a service is crash looping. It does not tell you why. Tares keeps the evidence for the why on the same timeline as the alert.

• The alert. Datadog posts the monitor event to a webhook source in Tares. Datadog reaches Tares through the webhook connector and a Datadog webhook setup. You write the payload as JSON from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so the shape is yours.
• The service’s metrics. A metrics source, for example Prometheus, carries the same service label. The agent sees the numbers around the moment the monitor fired.
• The service’s logs. A logs source, such as Docker container logs or OTLP, carries the same service label. The agent reads the log lines from the same window.
• Recent changes. A second webhook source records deploys, and another can record config changes. Most real root causes are a change, so this is the source that turns “the service is crash looping” into “a deploy landed at 14:02 and the errors started at 14:04”.

Because every source is keyed by the same service label, Tares merges them into one timeline per service on one clock. The agent reads that timeline in a single call and ties its diagnosis to specific evidence lines. It then writes its conclusion back as a finding on the timeline, so the next investigation starts from it. The full pattern is in Build an AI SRE.

Set it up

1. Create a webhook source for the Datadog monitors. In the console, go to Sources, Add source, webhook. Declare a label for each payload field you want to correlate on, with one primary. Use a service label so the alert lands on the right timeline.

- name: datadog-monitors
  connector: webhook
  config:
    event_type: datadog_monitor
    labels:
      - { name: service, field: service, primary: true }

2. Point Datadog at the ingest URL. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as flat JSON from Datadog variables, and include a service field. Keep the payload flat, because event_type_field and text_template read top-level fields only.
3. Add authentication if Tares has auth on. In the webhook’s custom headers, add an Authorization Bearer header holding an ingest-scoped API key. On an instance without auth you can leave the header out.
4. Call the webhook from the monitor. Put @webhook- followed by your webhook name in the monitor message so the monitor calls it.
5. Add the metrics and logs sources for the same service. Use the same service label on each, as the guide describes for Prometheus, Docker logs and OTLP.
6. Add a deploy source. Create another webhook source with a primary service label and a fixed event type of deploy. Then add one line to your CD pipeline that posts on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

7. Verify. POST a test payload and check Recent events on the source page. Fields shows which payload fields arrive and how often.
8. Connect an agent and ask. Start the MCP endpoint with tares mcp, connect your client, and ask what happened to the crash looping service in the last 15 minutes. The client answers from the correlated read.

What you get

• A Datadog monitor alert that sits on the service’s timeline next to its metrics, logs and deploys, not in a separate tool.
• A crash loop diagnosis that names the likely cause and cites the evidence lines behind it, such as a deploy followed by errors.
• A finding written back to the timeline, so the next investigation starts from the previous conclusion instead of from scratch.
• One read for you or any connected agent, and a trigger over your sources that can wake a Tares agent, your own agent service or a Slack channel.
• Datadog keeps owning the monitoring. Tares ingests what fires and correlates it with the evidence.

When something else fits

• If you only want the Datadog alert forwarded somewhere, without correlating metrics, logs and deploys, a plain webhook receiver is enough.
• If your alerts come from Prometheus rules or Alertmanager, use the prometheus_alerts or alertmanager connector, which Tares supports directly, instead of the Datadog webhook route.

Related