How do I alert on output tokens for one step?

Rius lets you alert on output tokens for one step with a custom alert that has a Step filter and a rule on the Output tokens metric. The Step filter matches the span name exactly, and it measures tokens only when that span is itself a model call. A step that wraps model calls, such as a chain, tool or agent span, carries no tokens of its own. For that case, filter on Model or on Span kind LLM instead.

How it works

A custom alert is one or more rules, combined with AND, evaluated over one shared rolling window. Filters decide which spans feed the rules, before any aggregation.

• The Step filter matches the span name exactly. It offers =, ≠, in and not_in, and suggests values from your workspace’s live data.
• Output tokens is a numeric metric, with the aggregations avg, min, max, p50, p95 and p99.
• Tokens sit on model-call spans (LLM, embedding, reranker). The rules measure only the spans your filters match, so a Step filter that selects a tool, chain or agent span measures no tokens. No filter measures the tokens of the model calls inside a step.
• Aggregation is per trace. Output tokens is not one of the per-call metrics, so avg(Output tokens) is the average per trace in the window, not the workspace total. Only Time to first token, Context usage and LLM calls are measured per LLM call.

So the step you pick must be the name of the model-call span. If you want to cover every model call regardless of step, filter Model, or filter Span kind to LLM. Add Embedding and Reranker with in if your agent calls them. To measure whole runs, filter Service or Agent.

Set it up

1. Open a trace that contains the step and copy the span name exactly as it appears. The match is exact, so a different name or an older SDK’s name will not match. Unnamed tool, agent, retrieval and model spans from the 1.0.0 and later SDKs are named after their operation and target, such as execute_tool search_orders.
2. In the custom alert builder, add a Step filter set to that span name. To cover several steps, use in with a list of names. An alert takes up to 10 filters with up to 50 values each.
3. Add a rule on Output tokens. Pick an aggregation, a comparison operator and a threshold in tokens. Use p95 or max to catch a few runaway generations, or avg to catch a shift in typical output length.
4. Set the window, from 5 minutes to 24 hours. All rules share it.
5. Attach a channel (webhook, email or Slack) and save. The builder shows a plain-language preview of when the alert fires, and that sentence becomes the alert’s description.

The alert reads like this:

FILTERS   Step = your-step-name
RULES     p95(Output tokens) > 2000
WINDOW    15 minutes

Rules combine with AND only. For either-or behaviour, such as one threshold on p95 and another on max, create a second alert. The general behaviour of alerts is on the Alerts page.

What you get

• An alert that fires when output tokens on the step you chose cross your threshold within the window.
• Firings that record the observed value of each rule against its threshold, so you can see which condition tripped.
• A description generated from your rule, shown on the alert’s details page.
• A clear boundary on what the Step filter covers: model-call spans named exactly as you set, not the model calls nested under a wrapper step.

When something else fits

• If the step is a wrapper such as a chain, tool or agent span, filter on Model or Span kind LLM to alert on model-call output tokens, or on Service or Agent to measure whole runs.
• If you need either-or conditions, create separate alerts, because rules combine with AND and there is no nesting.

Related