What should an AI SRE read next to Datadog monitors for 5xx errors on an API?

Tares gives an AI SRE the same service’s metrics, logs and recent deploys on one timeline beside the Datadog monitor alert. The Datadog alert reaches Tares through the webhook connector and a Datadog-side webhook setup. The agent then cites evidence lines in its diagnosis instead of reacting to the alert alone. Tares has no connector of its own for Datadog today.

How it works

A monitor firing tells you that something is wrong. The cause sits in the neighbouring signals, so Tares puts them on one timeline per service. Every source carries the same service label, and Tares merges what they send onto one clock.

For 5xx errors on an API, the agent reads these next to the monitor:

• Metrics: the 5xx rate, p99 latency, database connections and dependency health, from the prometheus connector.
• Logs: the API’s request logs, from the docker_logs connector or from OpenTelemetry through the otlp connector.
• Recent changes: deploy and config events, posted to a webhook source by your CD pipeline.
• The Datadog alert: delivered by a Datadog webhook to a Tares webhook source.

A deploy event that lands at 14:02 next to errors that start at 14:04 is the kind of evidence the agent can cite. Tares handles ingestion, per-service correlation, triggers and the MCP read surface. You bring the agent, or use a Tares agent with a prompt you edit. The agent reads the correlated timeline, names the likely cause, and writes a finding back onto the service’s timeline. The next investigation starts from that finding.

Set it up

1. Create a webhook source for the Datadog alerts. Use Sources → Add source → webhook, or ask a connected agent to create it. Declare a label per payload field you want to correlate on, and mark one as primary. Use the same service label name as your metrics and logs sources.

- name: datadog-monitors
  connector: webhook
  config:
    event_type: datadog_alert
    labels:
      - { name: service, field: service, primary: true }

2. Copy the ingest URL from the source page. In Datadog, open the Webhooks integration and add a webhook whose URL is that Tares ingest URL.
3. Write the Datadog payload as flat JSON from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK. Include a service field so the alert lands on the right timeline. Keep the payload flat, because event_type_field and text_template read top-level fields only.
4. If Tares has auth on, add a custom header to the Datadog webhook. It is an Authorization Bearer header that holds an ingest-scoped API key. Without auth on, you need no header.
5. Put @webhook- followed by the name you gave the webhook in the monitor message, so the monitor calls it.
6. Add the neighbouring evidence, keyed by the same service label. Point a prometheus source at your 5xx metrics, and a docker_logs or otlp source at your API logs.
7. Add a second webhook source for deploys, with a primary service label and event_type: deploy. Run one line in your CD pipeline on every deploy:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

8. Add a trigger over these sources, keyed by service, so a Datadog alert event wakes your agent. Start Tares with tares up, and run tares mcp if you want a connected client to investigate on demand.

POST a test payload and check Recent events on the source page. Fields shows which payload fields arrive. The webhook connector page has the full configuration, and the AI SRE guide walks through the metrics, logs, trigger and agent wiring end to end.

What you get

• One timeline for the API service. The Datadog monitor alert, the 5xx metric, the request logs and the latest deploy all sit on one clock.
• A diagnosis that points at specific evidence, such as the error log lines and the deploy that landed minutes before the spike.
• A finding written back to the timeline, so a connected MCP client asking what happened to the API reads the evidence and the earlier conclusion.
• Datadog stays where your alerting lives. Tares ingests what fired, correlates it, and wakes the agent. Nothing is re-thresholded.

When something else fits

• If your alerts come from Prometheus rules, the prometheus_alerts connector polls them with no vendor-side webhook setup. Use it when Prometheus owns your alerting instead of Datadog.
• If you only need a quick look at one monitor, the alert payload alone may be enough. The correlated timeline pays off when the cause is a change or a dependency.

Related