What trigger condition catches a disk filling up early?

Tares catches a disk filling up with a trigger that aggregates a numeric usage field over a short window, per host, and fires when it crosses a threshold set below the danger line. Use aggregate: max on the usage value, a predicate such as > 80, group_by: [key_value] so each host fires on its own, and a cooldown so a flapping disk does not page you repeatedly. Tares then wakes your agent with the host’s correlated timeline.

How it works

A trigger watches one or more sources and checks its condition as events arrive. For a disk, the condition has four parts:

• A typed field. Triggers aggregate a numeric field, so one of the trigger’s sources must produce one. A Prometheus source can carry the disk usage metric you pick by name, with a label such as { name: value, field: value, type: number } so the metric value is a number you can aggregate in triggers.
• An aggregate over a window. aggregate is one of count, sum, avg, max, min or any. For disk usage, max over a window of a few minutes fires when any reading in the window is over the line. avg smooths out short spikes if your metric is noisy.
• A predicate. The predicate is a comparison such as > 80. Choose the number in the unit your metric reports, and set it below the level where the disk actually runs out. That gap is what makes the alert early.
• Per-entity evaluation. group_by: [key_value] evaluates the condition per entity. key_field is the label that names the entity across your sources, here the host, so each host fires independently.

The cooldown is the minimum gap between firings for the same entity. A disk that hovers around the threshold fires once per cooldown, not on every reading.

When the condition holds, Tares POSTs one JSON body to each subscriber. The payload is the host’s correlated timeline across the trigger’s sources over context_window, so the agent sees the metrics and logs around the climb without a follow-up call.

Set it up

1. Add a source with a numeric usage field. Use the Prometheus connector, pick your disk usage metric by name, and give the source a label that names the host (mark it primary) plus a numeric value label. Name the source host_metrics to match the trigger below.
2. Add the trigger. In the console, open the project page and use + next to Triggers, or go to Triggers → Add trigger. You can also put it in the catalog YAML:

triggers:
  - name: disk_filling
    sources: [host_metrics]
    key_field: host
    condition:
      aggregate: max
      field: value
      predicate: "> 80"       # pick a level below the point where the disk is full
      window: 10m
      group_by: [key_value]   # evaluated per entity (per host)
    emit:
      kind: disk_filling
      context_window: 15m     # how much timeline to hand the agent
    cooldown: 1h              # minimum gap between firings for the same host

3. Optionally add filters to count only some events. Each filter is field, op and value, with op one of eq, neq, contains, gt, lt, gte, lte. For example, keep only production hosts with an env label.
4. Subscribe an agent. A Tares agent is woken by the trigger in a project that wires it to that trigger, once you turn it on there. An external agent registers a webhook over MCP with the subscribe tool:

Use tares to subscribe to the disk_filling trigger and deliver to https://my-agent.example.com/hook.

5. Check the result. The trigger’s page shows the condition, every subscriber and its delivery counts. Every firing is logged under Firings, even with zero subscribers.

An agent can create the same trigger over MCP with create_trigger and update_trigger, which take the same fields. The Watch with triggers guide covers every option in detail.

What you get

• An alert that fires while there is still room on the disk, because you set the predicate below the full mark.
• One firing per host, so a single filling disk does not hide behind a healthy fleet.
• A quiet channel. The cooldown limits repeats for the same host.
• An agent that starts with context. The dispatch carries the host’s timeline across the trigger’s sources over context_window.
• A record of every firing and delivery. Delivery is at-least-once, so dedupe on dispatch_id. Your endpoint acknowledges with a 2xx. 5xx responses and transport errors are retried with backoff, and a 4xx is recorded as failed and not retried.

For the wider picture of how triggers feed an investigating agent, see AI SRE.

When something else fits

• If you want to look at the disk on a clock instead of on a threshold, give the trigger a schedule with condition: {every: 10m}. It fires once per interval and hands over a summary of the window.
• If the condition needs more than one level, for example a warning at one threshold and an escalation at a higher one, define two triggers with different predicates and cooldowns.
• If Prometheus already has alert rules for the disk, you can watch the alerts those rules fire instead of the raw metric.

Related