What should Datadog monitors about 5xx errors on an API include for an AI SRE?

Tares works best when a Datadog 5xx monitor sends a flat JSON payload with a service name, a severity, a summary and a start time. The service name matters most, because it ties the alert to the metrics and logs of the same service on one timeline. Datadog reaches Tares through the webhook connector and a vendor setup, and you write the payload yourself.

How it works

Tares has no connector of its own for Datadog today. Datadog reaches Tares through the webhook connector and a vendor setup. A Datadog webhook posts JSON to the ingest URL of a Tares webhook source. You write that JSON in the webhook Payload field, using Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so the shape is yours to design.

The design goal is correlation. In an AI SRE setup, every source carries the same service label, and Tares merges metrics, logs and alerts for that service on one clock. The agent reads that timeline and ties the failure to specific evidence lines. An alert only joins the timeline if its payload names the service the same way your other sources do.

So a 5xx monitor should carry four things:

• Service name. The value must match the service label on your metrics and log sources, for example the name of the API. Mark it as the primary label on the webhook source.
• Summary. A short sentence about what broke, such as 5xx rate above threshold on the API. A text_template renders it as the event text.
• Start time. An ISO-8601 timestamp. Point event_time_field at it so the alert lands at the moment the errors began, next to the error logs and the metric that tripped it.
• Severity. A field such as critical or warning, declared as a label so you can see and correlate on it.

Keep the payload flat. event_type_field and text_template read top-level fields only, so nested objects are stored but cannot drive the event type or the text.

Set it up

1. Create a webhook source in Tares under Sources, Add source, webhook, or ask a connected agent to create it. Declare a label per payload field, with service as primary. Save, then copy the ingest endpoint from the source page.

- name: datadog-api-5xx
  connector: webhook
  config:
    event_type: alert
    event_time_field: started_at
    text_template: '{summary}'
    labels:
      - { name: service, field: service, primary: true }

2. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL.
3. Write the Payload as JSON with a flat set of fields: a service name, a severity, a summary and a start time. Fill the fields from Datadog variables such as $EVENT_MSG where they fit, and type fixed values, such as the API’s service name, where they do not. Use the same field names you declared in the Tares source.
4. If Tares has auth on, add custom headers as JSON on the Datadog webhook with an Authorization Bearer header holding an ingest-scoped API key. On an instance without auth you need no header.
5. Put @webhook- followed by your webhook name in the monitor message, so the monitor calls the webhook.
6. Check the mapping before relying on it. Post a sample payload with the same fields to the ingest URL, then open Recent events. Fields shows which payload fields arrive and how often.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"service": "api", "summary": "5xx rate above threshold", "started_at": "2024-01-01T00:00:00Z"}'

Add the severity field to your payload and declare it as a second label next to service the same way.

What you get

• Each Datadog 5xx alert lands on the timeline of the right service, next to the metrics and request logs for that service.
• The alert sits at its real start time, so the agent can compare when errors began with deploys and other changes.
• The event text is your summary, so the alert reads clearly in Explore and in the view the agent gets.
• A trigger watching the sources keyed by service can fire when the alert event arrives. A Tares agent or your own agent is then woken to take the first look.
• The agent writes its finding back onto the same service timeline, so the next investigation starts from the previous conclusion.

When something else fits

• If you run Prometheus rules and Alertmanager, the alertmanager connector gives you one event per alert with no payload design. Keep Datadog on the webhook when Datadog owns your alerting.
• If a payload field you need is nested, flatten it in the Datadog payload, because the event type and the text template read top-level fields only.

Related