How do I make a trigger take the max or average of a Prometheus metric’s value?

Tares stores each Prometheus metric value as a number-typed value label, so a trigger can run max, avg or sum on it. Filter the trigger to one metric by event_type, then aggregate value. Number labels are excluded from entity faceting, so value works as a measurement and not as a grouping key.

How it works

The Prometheus connector turns each series you pick into a metric event on every poll tick. The numeric reading is stored under a label named value, declared with type: number. Tares proposes that label for you automatically when you pick metrics.

Because the value is a number label, a trigger can aggregate it with max, avg or sum. Two details shape how you write the trigger:

• The metric name is the event type. When you pick metrics by name or by label, Tares compiles your picks into one selector with by_name: true, and each event is named by its own metric name. A filter on event_type therefore narrows the trigger to a single metric.
• Number labels are not facets. Tares leaves number labels out of entity faceting. The entity key comes from your primary label, such as service, and value is the thing you measure for that entity.

If you want a derived metric instead of a raw one, such as a rate or a p99, a queries entry can hold a single PromQL expression with an explicit event_type and field. The pick-metrics flow covers raw ingestion without any PromQL.

Set it up

1. Add the Prometheus source. In the console, go to Sources, Add source, Prometheus. Enter the url and any auth, choose Discover, then pick metrics by name or by a label they carry. You can also ask a connected agent to ingest the metrics you want. Either way you never write PromQL.
2. Confirm the value label. The generated config includes a number-typed value label next to your primary label:

- name: metrics
  connector: prometheus
  poll: 10s
  config:
    url: http://localhost:9090
    queries:
      - promql: '{__name__=~"(pg_stat_database_xact_commit|pg_up)"}'   # generated from your picks
        by_name: true
    labels:
      - { name: service, field: metric.service, primary: true }
      - { name: value, field: value, type: number }   # the metric value, aggregate it in triggers

3. Create the trigger on this source. Add a filter on the field event_type with the op eq and the metric name as the value, for example pg_stat_database_xact_commit.
4. Set the condition to aggregate value with max, avg or sum. Choose a predicate such as a threshold, and a window that sets how far back the aggregate looks.
5. Add a cooldown so the trigger does not fire again on every poll while the condition stays true.
6. Verify the data. Open the source and check Recent events for one event per series per poll. Check Fields for the labels such as metric.service and the numeric value. Pick the entity the series is keyed on in Explore to watch the metric move.

For how this fits into monitoring and escalation, see AI SRE.

What you get

• Threshold rules on real measurements, such as the highest value of a metric in a window, its average or its sum.
• A trigger scoped to the metric you care about through the event_type filter, so one metric’s readings are not mixed with another’s.
• Metrics that join with logs, deploys and alerts on the same entity key, because each series is keyed on your primary label.
• Setup without a query language. You pick metrics, and Tares builds the selector and proposes the value label.

When something else fits

• If you want to react to alerts that Prometheus rules already fire, use the prometheus_alerts connector, which polls firing and resolved alerts instead of raw values.
• If you need a derived number such as a rate or a p99, use a computed series with an explicit PromQL expression, event_type and field, and aggregate that field in the trigger.

Related