What should Datadog monitors about high latency include for an AI SRE?

A Datadog monitor about high latency should send Tares a flat JSON payload with a service name, a severity, a summary and a start time. Tares has no connector of its own for Datadog today. The monitor reaches Tares through the webhook connector and a Datadog webhook setup. With those four fields, the agent ties the alert to the metrics and logs of the same service.

How it works

Datadog calls a webhook you define, and you write the payload yourself as JSON in the webhook Payload field. You build it from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so the shape is yours. Tares stores the payload lossless in a webhook source and maps the fields you declare as labels into the event envelope.

Four things are worth sending for a high latency monitor:

• Service name. This is the field that matters most. Declare it as the primary label. Tares builds one timeline per service, so an alert whose service value matches the service label on your metrics and logs sources lands on the same timeline as that evidence. Use the same spelling everywhere.
• Severity. It tells the agent how urgent the latency breach is, and you can filter on it in triggers.
• Summary. Send the monitor’s message text, for example through $EVENT_MSG. Name the latency condition in the monitor message, so the summary reads as a clear statement of what crossed which threshold.
• Start time. Send an ISO-8601 timestamp and point event_time_field at it. The event then sits at the moment latency began, not at the moment the webhook arrived. This lets the agent compare it against deploys and error logs on the same clock.

Also send $ALERT_ID and $LINK. They are not needed for correlation, but they let the agent and your team trace a finding back to the Datadog monitor.

Keep the payload flat. event_type_field and text_template read top-level fields only, so a nested object hides fields from them.

Set it up

1. Create a webhook source in Tares under Sources, Add source, webhook, or ask a connected agent to create it. Declare a label for each field you send and mark the service label as primary. Save, then copy the ingest endpoint from the source page.

- name: datadog-latency
  connector: webhook
  config:
    event_type_field: action
    text_template: '{service} {summary}'
    event_time_field: started_at
    labels:
      - { name: service, field: service, primary: true }
      - { name: severity, field: severity }

2. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL.
3. Write the payload as flat JSON in the Payload field. Give it a service name, a severity, a summary from the monitor message and a start time, using the field names you declared as labels. Add the Datadog variables for the alert ID and link if you want the trace back.
4. If your Tares instance has auth on, add custom headers as JSON in the webhook. The header is an Authorization Bearer header holding an ingest-scoped API key. On an instance without auth you can skip it.
5. Put @webhook- followed by your webhook name in the monitor message, so the high latency monitor calls it.
6. Check the label mapping before you rely on it. Ask a connected agent to run test_source with a sample payload, or post one yourself with the shape below, then open Recent events and Fields on the source page. Replace the body with your own fields.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "deploy", "sender": "ci", "pipeline": "web"}'

7. Make sure a trigger watches this source and your metrics and logs sources, keyed by service, as described in AI SRE. When the latency alert arrives, the trigger wakes the agent with one correlated timeline.

What you get

• A latency alert that lands on the right service timeline, next to the metrics and logs for that service, instead of floating as an isolated message.
• An event stamped with the time latency started, so the agent can line it up against a deploy or a config change that landed minutes earlier.
• A readable summary and a severity, so the agent’s first look states what broke, how urgent it is and which evidence lines support the conclusion.
• A link back to the Datadog monitor when you send $LINK, and a stable identifier when you send $ALERT_ID.
• The full payload stored lossless, so a field you did not map as a label is still there to read later.

When something else fits

• If you already run Prometheus rules or Alertmanager for latency, the Prometheus alerts and Alertmanager connectors fit better, because they need no hand-written payload.
• If you want latency as a metric value rather than as an alert, poll the metric directly and threshold it in a trigger.
• Datadog is a good source when its monitors already hold your alerting logic and you only want Tares to correlate what fires.

Related