How do I alert on reasoning tokens for one step?

In Rius, create a custom alert with a Step filter set to the exact span name of the model call, then add a rule on the Reasoning tokens metric, such as avg(Reasoning tokens) above your limit. The Step filter selects which spans feed the rule before anything is aggregated. Reasoning tokens are aggregated per trace, so the value is the average per trace in the window, not a workspace total. The step you pick must be the span of the model call itself, because tokens sit on model-call spans.

How it works

A custom alert is one or more rules, combined with AND, evaluated over one shared rolling window. Filters narrow which spans feed those rules, and they apply before aggregation.

• The Step filter matches the span name exactly. Values are suggested from your workspace’s live data, so you can pick the name from a list instead of typing it.
• Where the tokens are. Input, output and reasoning tokens are carried by model-call spans. If the Step you filter on is a tool, chain or agent span, the rule measures no tokens. No filter measures the tokens of the model calls inside one step. To alert on one step’s reasoning tokens, filter on the name of the model-call span.
• Per-trace aggregation. Reasoning tokens is a numeric metric. It offers avg, min, max, p50, p95 and p99, and these run per trace: avg(Reasoning tokens) is the average per trace in the window. Time to first token, Context usage and LLM calls are the exception, because they are measured per LLM call, so their aggregations run over calls.
• Other filters. You can add a Model or Span kind filter next to Step to narrow the match further. Text fields offer =, ≠, in and not_in.

The builder shows a plain-language preview of when the alert fires, and that sentence becomes the alert’s description. The full list of fields and metrics is in Custom alerts.

Set it up

1. Open the alert builder and create a custom alert. For the wider alerting model, see Alerts.
2. Add a filter on Step, choose =, and pick the span name of the model call from the suggested values. Use in if the same model call appears under more than one name.
3. Optionally add a Model or Span kind filter to narrow the match further.
4. Add a rule on Reasoning tokens, choose an aggregation such as avg or p95, a comparison operator (<, <=, >, >= or ==) and a token threshold.
5. Set the window. It is shared by all rules and runs from 5 minutes to 24 hours.
6. Read the preview sentence to confirm it describes what you want, then save and attach a channel.

The alert you build reads like this:

FILTERS   Step = your-model-call-span-name
RULES     avg(Reasoning tokens) > 2000
WINDOW    15 minutes

The step name and the threshold are examples. Use your own span name and a value that fits your traffic. An alert takes up to 10 filters with up to 50 values each, and up to 10 rules.

If you upgrade to the 1.0.0 SDKs, unnamed model, tool, agent and retrieval spans are named after their operation and target, such as execute_tool search_orders. A Step filter written against the old name then needs the new one.

What you get

• An alert that fires on reasoning-token usage from one named model call instead of from every call in the service.
• A value you can read directly: with avg it is the average per trace in the window, and with p95 or p99 it shows the heavier traces.
• Firings that record the observed value of every rule against its threshold, so you can see which condition tripped.
• A details page with the generated description, the editable rule, attached channels and firings.
• The option to add a second rule with AND, for example a trace-count rule, so the alert fires only when both conditions hold. Rules combine with AND only, so for either-or behavior create a second alert.

When something else fits

• If you want reasoning tokens across whole runs, filter on Service or Agent instead of Step, so the rule covers every model call in the run.
• If you need model-call behavior such as context usage or time to first token, those metrics are measured per LLM call, so their aggregations run over calls, not traces.
• If you need either-or conditions, create separate alerts, since a single alert supports AND only.

Related