What should an AI SRE read next to Datadog monitors for a disk filling up?

Tares puts the Datadog monitor alert, the same service’s metrics, its logs and its recent deploys on one timeline, so an AI SRE reads all of them together when a disk is filling up. Datadog reaches Tares through the webhook connector and a Datadog setup, and Tares has no connector of its own for Datadog today. Every source carries the same service label, so the diagnosis cites evidence lines that sit beside the alert.

How it works

The monitor alert tells you a disk is filling. It does not tell you why. The agent needs three more things from the same service, and Tares keeps them together:

• The Datadog alert. A Datadog webhook posts to a Tares ingest URL. You write the payload in the Datadog webhook Payload field from Datadog variables such as $ALERT_ID, $EVENT_MSG and $LINK, so the shape is yours. Tares stores it lossless as a webhook source.
• The service’s metrics. A prometheus source polls the series you pick by name or label, so you can pick the disk usage series for that service and read how fast it grows. No PromQL is needed.
• The service’s logs. A docker_logs source tails the container’s logs, or an otlp source receives them from your OpenTelemetry pipeline. Log lines show what was writing when the disk grew.
• Recent changes. A second webhook source receives a deploy event from your CD pipeline on every deploy. Most root causes are a change, so this is the signal that turns a symptom into a cause. A further webhook source with event_type: config covers config changes the same way.

Because all of these sources carry the same service label, Tares builds one timeline per service. An agent reads it with one call, for example read with the service name, and sees the alert, the metric curve, the log lines and the deploy on one clock. A trigger can wake an agent when the alert event arrives. The agent writes its finding back onto the same timeline, so the next investigation starts from that conclusion. The full build is in AI SRE.

Set it up

1. Start Tares and the MCP endpoint your agent reads through.

tares up
tares mcp

2. Create a webhook source for Datadog in Sources, Add source, webhook, with a label for each payload field you want to correlate on and one marked primary. Keep the payload flat, because event_type_field and text_template read top-level fields only. Use the same service label name your metric and log sources use.

- name: datadog-monitors
  connector: webhook
  config:
    event_type: datadog_monitor
    labels:
      - { name: service, field: service, primary: true }

3. Copy the ingest endpoint from the source page. In Datadog, open the Webhooks integration and add a webhook whose URL is that Tares ingest URL. Write the payload as JSON from Datadog variables, including a service field that matches your label.
4. If Tares has auth on, add custom headers as JSON in the Datadog webhook with an Authorization Bearer header that holds an ingest-scoped API key. The header is the credential.
5. Put @webhook- followed by your webhook’s name in the disk monitor’s message, so the monitor calls it.
6. Add the evidence beside it. Create a prometheus source for the disk usage series, a docker_logs or otlp source for the logs, all labeled with the same service. Then add a deploy source and call it from your CD pipeline:

curl -X POST https://<your-tares>/ingest/<ingest_key> \
  -H 'content-type: application/json' \
  -d '{"service": "checkout", "commit": "9f2c81d", "author": "dana", "message": "raise db pool size"}'

7. Check Recent events on the Datadog source, and Fields to see which payload fields arrive. Details are on the webhook connector page.

What you get

• A disk alert that arrives with context. The monitor, the usage curve, the log lines and the last deploy for that service are on one timeline.
• A diagnosis that cites evidence. The agent can point to the deploy that landed before the growth started or the log lines written during it, instead of restating the alert.
• A finding that stays. The agent writes its conclusion back to the service’s timeline, and the next investigation, human or agent, starts from it.
• A choice of who gets woken. A trigger can wake a Tares agent, your own agent service over a webhook, or a Slack channel.
• Datadog stays where it is. It keeps owning the monitor, and Tares ingests what it fires and correlates it.

When something else fits

If the disk alert is the only signal you have and no other source shares its service label, the agent has nothing to cite beyond the alert, so add the metric, log and deploy sources first. If your alerts already come from Prometheus rules, the prometheus_alerts or alertmanager connectors take them directly and you do not need the Datadog webhook for that path.

Related