What should Datadog monitors about crash loops include for an AI SRE?

Tares gets the most from a Datadog crash loop monitor when the webhook payload carries a service name, a severity, a summary and a start time. The service name ties the alert to that service’s timeline of metrics and logs, and the other three tell the agent what fired and when. Datadog reaches Tares through the webhook connector and a vendor setup, and Tares has no connector of its own for Datadog today.

How it works

Datadog calls a webhook you define, and you write the JSON payload yourself in the webhook Payload field from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK. Because the shape is yours, you decide which fields the AI SRE sees. Tares stores the whole payload lossless through the webhook connector, and labels map payload fields into the event so Tares can correlate it.

For a crash loop alert, send these four things:

• Service name. This is the field Tares correlates on. Declare it as the primary label, and use the same label name on your metric and log sources. Then the crash loop alert lands on the same per-service timeline as the evidence the agent reads. See Build an AI SRE for how one timeline per service works.
• Severity. It tells a trigger or the agent how urgent the alert is.
• Summary. One line on what the monitor saw, for example that a pod keeps restarting. Use the Datadog variable that carries the monitor message, such as $EVENT_MSG. Tares can render it as the event text with a text template.
• Start time. An ISO-8601 timestamp of when the alert began. Point the source’s event time field at it, so the alert sits at the right moment on the timeline next to the first error logs.

Keep the payload flat. The event type field and the text template read top-level fields only, so put these four fields at the top level of the JSON, not nested. $LINK is a useful extra field, because it gives the agent a way back to the Datadog monitor.

Set it up

1. In Tares, create a webhook source (Sources → Add source → webhook) or ask a connected agent to create it. Declare a primary label for the service name and a label per other field you want to correlate on. Copy the ingest URL from the source page.

- name: datadog-crash-loops
  connector: webhook
  config:
    text_template: '{summary}'
    event_time_field: started_at
    labels:
      - { name: service, field: service, primary: true }

2. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL.
3. Write the payload as flat JSON from Datadog variables, with the service name, severity, summary and start time at the top level. Include $LINK if you want a pointer back to the monitor.
4. If Tares has auth on, add custom headers as JSON with an Authorization Bearer header holding an ingest-scoped API key. On a secured instance that header is the credential. Without auth you can leave the header out.
5. Put the @webhook handle for your webhook’s name in the crash loop monitor message, so the monitor calls it.
6. Check the mapping with a sample payload. You can ask a connected agent to run test_source, or post one yourself:

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"service": "checkout", "severity": "critical", "summary": "checkout pods are crash looping", "started_at": "2024-01-01T00:00:00Z"}'

7. Open Recent events on the source. Fields shows which payload fields arrive and how often, so you can confirm the service name is present on every event and pick a reliable key.

Then add a trigger over the webhook source and your metric and log sources, all keyed by the same service label, so a crash loop alert wakes the agent with the correlated timeline.

What you get

• Each crash loop alert lands as one event on the timeline of the service it names, next to the metrics and logs for that service.
• The agent reads what fired, how severe it is and when it started, and can tie the alert to specific evidence lines, such as the first error logs after the start time.
• The summary appears as the event text, so the alert reads clearly in Explore and in the timeline an agent receives.
• The full payload is stored lossless, so fields you did not label are still there.
• A trigger can wake an agent when the alert arrives, and the agent can write its finding back to the same timeline for the next investigation. Datadog keeps owning the alerting, and Tares correlates and wakes.

When something else fits

If your crash loop alerts come from Prometheus rules or Alertmanager, the prometheus alerts and alertmanager connectors fit better, because they carry labels such as severity without a payload you write by hand. If the monitor cannot set custom headers or write a payload, point it at a different route, such as OTLP or a polled http_poll source.

Related