What should an AI SRE read next to Datadog monitors for high latency?

Tares puts the Datadog monitor alert on one timeline with the same service’s metrics, logs and recent deploys, so the AI SRE reads the alert and the evidence around it together. Datadog reaches Tares through the webhook connector and a vendor setup, and every source carries the same service label. The agent then cites the p99 metric, the error logs and the deploy that landed just before the alert in its diagnosis.

How it works

A monitor firing tells you that latency is high. It does not tell you why. Most real causes are a change, a dependency or a saturated resource, and the evidence for each sits in other systems.

Tares correlates by label. Every source that carries the same service label lands on one timeline for that service, with metrics, logs, alerts and changes merged on one clock. An agent reads that single timeline instead of querying each system.

For a high latency alert, put these signals next to the Datadog monitor:

• Metrics for the service. Use the Prometheus connector for p99 latency, 5xx rate and database connections, with service read from the metric labels.
• Logs for the service. Use the Docker logs connector for per-container request logs, or OTLP if you run an OpenTelemetry pipeline.
• Deploys and config changes. Use a second webhook source that your CD pipeline posts to on every deploy. This is the highest-value addition, because a line like “deploy landed 14:02, latency rose 14:04” is often the whole answer.
• The Datadog alert itself. Tares has no connector of its own for Datadog today. The monitor reaches Tares through the webhook connector, which stores the payload lossless.

The key is the shared label. If the Datadog payload names the service the same way your metrics and logs do, the alert lands on the same timeline as the evidence.

Set it up

1. Start Tares with tares up, then open Sources → Add source → webhook in the console. Declare a label for the service field in your payload and mark it primary. Keep the payload flat, because event_type_field and text_template read top-level fields only.

   - name: ci-events
     connector: webhook
     config:
       event_type_field: action
       text_template: '{action} by {sender}'
       labels:
         - { name: pipeline, field: pipeline, primary: true }

   For the Datadog source, use the same shape, but make the label { name: service, field: service, primary: true } and name the source for your monitors. Save it and copy the ingest endpoint from the source page.

2. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL.

3. Write the Payload field as JSON built from Datadog variables. Include a top-level service field so the alert carries the label you declared. If Tares has auth on, add custom headers as JSON with an Authorization Bearer header holding an ingest-scoped API key. On a secured instance that header is the credential.

4. Put @webhook- followed by your webhook name in the monitor message, so the high latency monitor calls it.

5. Add the deploy source. Create another webhook source with a primary service label, then add one line to your CD pipeline, run on every deploy:

   curl -X POST https://<your-tares>/ingest/<ingest_key> \
     -H 'content-type: application/json' \
     -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

6. Add your metrics and logs sources keyed by the same service label. Then open Explore, pick the service, and check that the alert, the p99 metric, the logs and the deploy sit on one timeline. Recent events on the webhook source and its Fields view show which payload fields arrive.

7. Build a trigger over those sources, keyed by service. You can wake an agent when an alert event arrives, or threshold the typed fields directly, for example max(p99_ms) > 1000 for latency. The full walkthrough is in Build an AI SRE.

What you get

• A diagnosis that cites evidence. The agent ties the latency alert to specific metric points, log lines and a named deploy instead of repeating the alert text.
• One read instead of many. The agent gets the correlated timeline for the service, not a separate query per system.
• A starting point that carries over. The agent writes its finding back onto the service’s timeline, so the next investigation, yours or an agent’s, starts from that conclusion.
• Your alerting stays where it is. Datadog keeps owning the monitor and the threshold, and Tares ingests what fired, correlates it with the evidence and handles the waking.
• A deploy trail. Every POST from your pipeline lands as a deploy event on that service, so “what changed” is always next to “what broke”.

When something else fits

• If the monitor is the only signal you have and nothing else is wired up, the alert alone gives an agent little to cite. Add at least the service’s metrics or deploy events first.
• If your payload cannot carry a service name that matches your other sources, the alert lands on a different timeline from the evidence. Fix the label first, then rely on the correlated read.
• If your alerts come from Prometheus rules or Alertmanager rather than Datadog, use the dedicated Prometheus alerts or Alertmanager connectors instead of a webhook source.

Related