Which Prometheus metrics show a disk filling up?

Tares ingests the disk metrics you pick from your Prometheus server and stores each reading as a number-typed value label that a trigger can aggregate. You choose the metrics by name or by a label they carry, and Tares builds the selector, so you write no PromQL. Pick the metrics that report disk space on your server and have a trigger take the max or avg of value to catch a filling disk.

How it works

The prometheus connector polls your Prometheus server on an interval. Each series becomes one metric event per poll, keyed on the entity you choose, such as a service or a host, so it joins with logs, deploys and alerts on the same key.

• Picking by name. Type a metric name or a prefix_* pattern and tick the matches. Use this when your disk metrics share a prefix.
• Picking by label. Choose a label, see the metrics that carry it, and tick the ones you want. Use this when a label such as a mount point or device marks your disk series.
• Discovery. Given your server’s url, Tares lists its metrics and labels, so you see which disk metrics your Prometheus exposes before you pick.
• The value label. The reading is stored as a number-typed value label, proposed automatically. A trigger can run max, avg or sum over it.
• Event type. A basket of picked metrics is named by metric name, so each event carries its own metric name. A trigger filters on event_type to focus on one disk metric, then aggregates value.

The disk metric names come from your Prometheus server and the exporters feeding it, so discovery is where you read them. If your disk numbers arrive through OpenTelemetry instead, the OpenTelemetry connector receives OTLP metrics, and resource attributes such as service.name become labels.

Set it up

1. Ask a connected agent to do the work: “Use tares to ingest the disk metrics from the Prometheus at localhost:9090.” It runs discovery, samples your picks, tests one poll and creates the source. Or go to Sources → Add source → Prometheus in the console.
2. Enter the url of your Prometheus server, plus a bearer token or basic-auth username and password if it needs them, then choose Discover.
3. Pick the disk metrics by name, by prefix pattern or by label, and confirm. Tares proposes the entity key and labels from a sample.
4. Check the generated source. It looks like this, with your own metric names in the promql line:

- name: metrics
  connector: prometheus
  poll: 10s
  config:
    url: http://localhost:9090
    queries:
      - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
        by_name: true
    labels:
      - { name: service, field: metric.service, primary: true }
      - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

5. Add a trigger that filters to one disk metric and aggregates value. Replace the placeholders with your metric name and your own limit:

- name: disk-filling
  sources: [metrics]
  filters:
    - { field: event_type, op: eq, value: your_disk_metric_name }
  condition: { aggregate: max, predicate: "> your_limit", window: 5m }
  cooldown: 5m

6. Open the source and check Recent events for one event per series per poll. Fields shows the labels and the numeric value, and Explore lets you watch a metric move for the entity it is keyed on.

The full field list for the connector is on the Prometheus connector page.

What you get

• Disk readings from your own Prometheus, ingested without writing PromQL.
• One metric event per series per poll, keyed on the same entity as your logs, deploys and alerts, so a rising disk reading sits next to the deploy or log burst that may explain it.
• A number-typed value label that triggers aggregate with max, avg or sum over a window you set.
• A trigger that fires on one disk metric because you filter on event_type, with a cooldown so a sustained condition does not repeat.
• The same picture in AI SRE, where an agent can correlate the disk reading with the rest of the entity’s events.

If you need a computed figure, such as a derived rate, a queries row can hold a single PromQL expression with an explicit event_type and field. Use it only when a raw metric is not enough.

When something else fits

• If your disk numbers come from OpenTelemetry rather than a Prometheus server, use the OTLP connector, which receives metrics and turns resource attributes into labels.
• If you need a derived series, such as a rate of change, write a computed series row in the Prometheus source instead of picking raw metrics.

Related