What should an AI SRE read next to Datadog monitors for slow database queries?

Tares gives an AI SRE the same service’s metrics, logs and recent deploys on one timeline beside the Datadog monitor alert for slow database queries. The Datadog alert reaches Tares through the webhook connector and a Datadog webhook. You label every source with the same service, so the agent reads one correlated timeline and cites the evidence lines behind its diagnosis.

How it works

A monitor firing is only the start of an investigation. The cause usually sits in the evidence around it: a latency metric that climbed, database connection counts, error logs, or a deploy that landed minutes earlier.

Tares keeps all of that on one timeline per service. Every source that carries the same service label merges onto one clock. The agent reads that single timeline instead of querying each system itself.

For slow database queries, put these next to the Datadog alert:

• Metrics for the service. Query latency (p99), database connections and dependency health, for example from the Prometheus connector.
• Logs for the service. Request and application logs, from the Docker logs connector or from OTLP.
• Recent changes. Deploys and config changes, posted to a webhook source from your CD pipeline. Most real root causes are a change, so the agent sees “deploy landed 14:02, queries slowed at 14:04” directly.
• Earlier findings. Conclusions that earlier investigations wrote back to the same timeline, so the next one starts from them.

Tares has no connector of its own for Datadog today. The monitor alert reaches Tares through the webhook connector and a Datadog webhook setup. Datadog posts a JSON payload that you write yourself in the webhook Payload field, and Tares stores it lossless as an event on the service’s timeline. The metrics, logs and deploys come from the other sources you label with the same service.

Set it up

1. Create a webhook source for the monitors. In the console, go to Sources, Add source, webhook. Mark one label as primary. Keep the payload flat, because event_type_field and text_template read top-level fields only.

- name: ci-events
  connector: webhook
  config:
    event_type_field: action
    text_template: '{action} by {sender}'
    labels:
      - { name: pipeline, field: pipeline, primary: true }

Use this shape for the Datadog source: a primary label named service that reads the payload’s service field, and a text_template that renders the payload’s message field. Save it and copy the ingest endpoint from the source page.

2. Check the endpoint. POST a test payload to /ingest/<ingest_key>, then look at Recent events and Fields on the source page.

curl -X POST http://localhost:8787/ingest/<ingest_key> \
  -H 'Content-Type: application/json' \
  -d '{"action": "deploy", "sender": "ci", "pipeline": "web"}'

3. Add the webhook in Datadog. Open the Webhooks integration and add a webhook whose URL is the Tares ingest URL. Write the payload as flat JSON. Write the service name as a literal in a service field, and set a message field to $EVENT_MSG. You can add other Datadog variables such as $ALERT_ID and $LINK as extra flat fields.

4. Authenticate if Tares has auth on. Add custom headers as JSON with an Authorization Bearer header that holds an ingest-scoped API key. On a secured instance that header is the credential.

5. Call the webhook from the monitor. Put @webhook- followed by your webhook name in the slow-query monitor’s message, so the monitor posts to Tares when it fires.

6. Label the evidence with the same service. Add your metrics, logs and deploy sources, each with a service label. For deploys, create a second webhook source and run one curl POST from your CD pipeline on every deploy. The AI SRE guide walks through all three sources and shows the deploy POST.

7. Wake an agent. Create a trigger over those sources, keyed by service, and subscribe a Tares agent, your own agent service or a Slack channel to it.

What you get

• The Datadog alert lands on the service’s timeline beside its metrics, logs and deploys, on one clock.
• The agent starts with that correlated read, so its diagnosis ties the slow queries to specific evidence lines, such as a latency rise, connection pressure or a deploy just before.
• The finding is written back to the timeline. The next investigation, whether your agent or a connected MCP client, reads it and starts from the previous conclusion.
• Datadog keeps owning the alerting. Tares ingests what fired, correlates it and handles the waking. Nothing is re-thresholded.

When something else fits

• If the only thing you need is a page when a monitor fires, Datadog’s own notifications are enough. Tares adds value when an agent has to explain the alert.
• Tares reads Datadog only through what the webhook payload carries. If you want the underlying metric series on the timeline, send them from a source Tares can read, such as Prometheus or OTLP, and label them with the same service.

Related