Which OpenTelemetry metric series show a disk filling up?

Tares receives the OpenTelemetry metric series your exporter sends over OTLP, labeled by service, so you can find your disk series in Recent events and watch them in Explore. One otlp source takes metrics, logs and traces from every service. Resource attributes such as service.name become labels, so a filling disk lines up with the same service’s logs and traces.

How it works

Tares has one connector for OpenTelemetry, called otlp. It receives OTLP/HTTP at POST /v1/metrics, alongside /v1/logs and /v1/traces. With the otlp-grpc extra it also receives OTLP/gRPC on port 4317.

• One source for every service. Resource attributes become labels. The service label comes from service.name, and it is the natural key for the source.
• Series come from your exporter. Which series carry disk usage depends on what your app, host agent or OpenTelemetry Collector sends. Tares stores the metric points it receives and shows them as they arrive, so you read the disk series names off your own data.
• No connector-specific fields. The otlp connector takes only labels. Resource attributes are available as label fields, for example service from service.name.
• Triggers watch the source. A trigger names its sources, filters events, and sets a condition with an aggregate, a predicate and a window, plus an optional cooldown. Use one to turn a disk series into an alert.

This page is part of the AI SRE guides, where triggers and agents act on what the sources collect.

Set it up

1. Start Tares locally.

   tares up

2. Create the otlp source. Ask a connected agent, “Use tares to create an OTLP source keyed by service”, or use Sources → Add source → otlp in the console. The source looks like this:

   - name: otel
     connector: otlp
     config:
       labels:
         - { name: service, field: service, primary: true }

3. Point your exporter at Tares. Use the first line for OTLP/HTTP, or the second for gRPC.

   OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8787   # OTLP/HTTP → /v1/{logs,traces,metrics}
   # or gRPC (otlp-grpc extra): OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

   The sender uses a Bearer Authorization header holding an ingest-scoped API key, needed only when Tares has auth on.

4. Open the source and look at Recent events. Metric points appear as they arrive, so you can spot the series your exporter uses for disk or filesystem usage and for capacity. In Explore, pick the service key to see which services are reporting.

5. Create a trigger on the otel source. Filter it to the disk series you found, and set a condition with an aggregate, a predicate and a window. A condition such as aggregate count, predicate ”> 0” and window 5m is the shape the triggers use. Add a cooldown so a disk that stays full does not alert on every point.

6. If your disk metrics are also exposed on a Prometheus server, you can ingest them with the Prometheus connector instead. You pick metrics by name or by a prefix_* pattern, and each series arrives with its numeric value stored as a number-typed value label. A trigger can then max, avg or sum that value, filtered to one metric by event_type.

What you get

• Every service’s OpenTelemetry metrics in one source, keyed by service, with no per-service setup.
• Disk metric points visible in Recent events as they arrive, so you confirm the series and labels your exporter really sends.
• A trigger that watches the disk series and fires on a condition you set, with a cooldown.
• Correlation by service, so a disk alert sits next to the logs and traces from the same service.
• A second route through the OTLP connector page and the Prometheus connector when your disk metrics live in Prometheus, where the value is a number a trigger can aggregate.

When something else fits

• If your disk metrics are already on a Prometheus server, the Prometheus connector is the better fit. You pick the metrics by name and the value is stored as a number-typed label for triggers to aggregate.
• If your exporter sends no disk or filesystem metrics, nothing in Tares can show a disk filling up. Add a host or Collector exporter that reports them first, then use the otlp source.

Related