How do I alert on reasoning tokens for one span kind?

In Rius, create a custom alert with a Span kind filter and a rule on the Reasoning tokens metric. Set the filter to LLM, because reasoning tokens sit on model-call spans, then pick an aggregation, a comparison and a threshold. Rius counts only the spans the filter matches, and it aggregates reasoning tokens per trace, not per call.

How it works

A custom alert is one or more rules combined with AND, evaluated over one shared rolling window. Filters run first and select which spans feed the rules, so the filter decides what Reasoning tokens covers.

• Span kind filter. It matches the kind of each span: LLM, tool, chain, retriever and so on. Tokens are recorded on model-call spans, so a filter that selects only tool, chain or agent spans measures no tokens. To alert on reasoning tokens, set Span kind to LLM. Embedding and reranker calls produce no reasoning tokens, so LLM is the kind that matters here.
• Per-trace aggregation. Reasoning tokens is a numeric metric with the aggregations avg, min, max, p50, p95 and p99. Numeric metrics use per-trace semantics: avg(Reasoning tokens) is the average per trace in the window, not a workspace total. p95(Reasoning tokens) fires on the heaviest 5% of traces, measured over the matched spans.
• Per-call metrics are the exception. Only Time to first token, Context usage and LLM calls are measured per LLM call, with aggregations over calls. Reasoning tokens is not one of them.
• Comparison operators. Use <, <=, >, >= or == against a token count.

The builder shows a plain-language preview of when the alert fires, and that sentence becomes the alert’s description on its details page.

Set it up

1. Create a custom alert. The full builder is described on Custom alerts.
2. Add a filter: field Span kind, operator =, value LLM. Text filters suggest values from your workspace’s live data.
3. Add a rule on Reasoning tokens. Choose an aggregation (for example p95), a comparison operator and a threshold in tokens. The threshold below is an example, use your own.
4. Set the window. It applies to all rules and runs from 5 minutes to 24 hours.
5. Attach a channel (webhook, email or Slack) and choose a Renotify policy, off or every N minutes.

The finished alert reads like this:

FILTERS   Span kind = LLM
RULES     p95(Reasoning tokens) > 4000
WINDOW    15 minutes

To narrow the alert further, add more filters, such as Service, Agent, Model or Session. An alert takes up to 10 filters with up to 50 values each, and up to 10 rules. Rules combine with AND only. For either-or behaviour, create a second alert.

What you get

• An alert that fires when reasoning tokens per trace cross your threshold, counting only the span kind you chose.
• Firings that record the observed value of every rule against its threshold, so you see which condition tripped.
• A generated description on the alert’s details page, taken from the builder’s preview sentence.
• A rule you can edit later, and an alert you can delete. Deleting removes its firing history after an explicit confirmation.

To see how this fits with the other alert types, read Alerts.

When something else fits

• If you want to measure whole runs rather than one kind of span, filter by Service or Agent instead of Span kind. With a Span kind, Step, Model or Operation filter, Duration runs from the earliest matched span’s start to the latest matched span’s end, not across the whole run.
• If you want one alert to fire on either of two conditions, create two alerts, because rules combine with AND only.

Related