What should Datadog monitors about a disk filling up include for an AI SRE?

Tares works best when each Datadog disk monitor sends a flat JSON payload with a service name, a severity, a summary and a start time. Datadog reaches Tares through the webhook connector and a Datadog setup, because Tares has no connector of its own for Datadog today. The service name ties the alert to the right timeline, and the summary and start time tell the agent what happened and when to look.

How it works

Datadog calls a webhook you define in its Webhooks integration. You write the payload yourself in the webhook Payload field, so you decide which fields the alert carries. Tares receives it on a webhook source and stores the payload lossless. Labels you declare map payload fields into the event so Tares can correlate on them.

For a disk that is filling up, four fields do the most work:

• Service name. This is the label the agent correlates on. Mark it primary, and use the same service name your metric and log sources use, so the disk alert lands on the same timeline as that service’s other evidence. The AI SRE guide keys every source by one shared service label for this reason.
• Severity. It tells the agent how urgent the alert is, so it can treat a warning differently from a critical.
• Summary. A short sentence from the monitor, such as the text Datadog puts in $EVENT_MSG. Name the host or mount in it if the monitor knows them. Tares can show it as the event text through text_template.
• Start time. An ISO-8601 timestamp in its own field. The webhook connector can read it as the event timestamp through event_time_field, and the field stays in the stored payload either way.

Keep the payload flat. event_type_field and text_template read top-level fields only, so nested objects cannot drive the event type or the text.

Set it up

1. In Tares, create a webhook source (Sources → Add source → webhook) with a primary label for the service and, if you want, one for severity. Copy the ingest endpoint from the source page. This is the shape to follow:

- name: ci-events
  connector: webhook
  config:
    event_type_field: action
    text_template: '{action} by {sender}'
    labels:
      - { name: pipeline, field: pipeline, primary: true }

Replace the example names with your own payload fields. For the disk monitor that means a service label as the primary one, a text_template over your summary field, and event_time_field set to your start-time field.

2. In Datadog, open the Webhooks integration and add a webhook whose URL is the Tares ingest URL, /ingest/<ingest_key> on your Tares address.

3. Write the payload as flat JSON from Datadog variables. Put the service name, the severity, the summary and the start time in their own top-level fields. Fill each from the Datadog variable that carries it, such as $EVENT_MSG for the summary.

4. If Tares has auth on, add custom headers as JSON in the Datadog webhook with an Authorization Bearer header that holds an ingest-scoped API key. On an instance without auth the header is not needed.

5. Put @webhook-<name> in the disk monitor’s message so the monitor calls the webhook.

6. Trigger the monitor or post a test payload, then open Recent events on the source page. Fields shows which payload fields arrive and how often, so you can confirm the service name is present on every event.

To check your label mapping before Datadog is involved, ask a connected agent to run test_source with a sample payload.

What you get

• Each disk alert arrives as an event labeled with the service, so it sits on that service’s timeline next to its metrics, logs and other alerts.
• The full payload is stored lossless, so the agent can read every field you sent, not only the labeled ones.
• The summary gives the agent a readable line for the event, and the severity tells it how serious the alert is.
• The start-time field is available in the payload, and with event_time_field set Tares reads it as the event timestamp.
• A trigger can watch the source and wake an agent when a disk alert arrives. The agent then reads one correlated timeline instead of starting from the Datadog alert alone.
• Fields on the source page shows how reliably each field arrives, which tells you whether the service name is a safe key.

When something else fits

• If the disk metric itself matters more than Datadog’s alert, a Prometheus source gives the agent the metric values to read next to the alert. That needs a Prometheus server, which Datadog does not replace.
• If your alerts already flow through Alertmanager, the alertmanager connector takes them directly and makes one event per alert.

Related