How do I alert on input tokens for one step?

Rius lets you alert on input tokens for one step by creating a custom alert with a Step filter and an Input tokens rule. The Step filter matches the span name exactly and narrows the spans before the rule aggregates them. You pick the aggregation, such as avg or p95, the threshold and the rolling window.

How it works

A custom alert is one or more rules combined with AND, all evaluated over one shared rolling window. Filters are optional and apply before any aggregation. Each filter matches a span’s own value, so only spans that match feed your rules.

Two parts of the alert matter for this question:

• The filter. Step matches the span name exactly. Use = for one step, or in to cover several step names in one alert. Values are suggested from your workspace’s live data, so you can pick the name instead of typing it.
• The metric. Input tokens is a numeric metric with the aggregations avg, min, max, p50, p95 and p99, in tokens.

Numeric aggregations use per-trace semantics. avg(Input tokens) is the average per trace in the window, not the workspace total and not a per-call average. Three metrics are measured per LLM call instead: Time to first token, Context usage and LLM calls. Their aggregations run over calls, not traces. Input tokens is not one of them.

Rules measure only the spans the filter matched. Input tokens and cost are recorded on LLM spans, the model calls, so name a step whose span is an LLM call. A Step filter on a tool, chain or agent span measures no tokens and no cost. To alert on model spend across a run, filter Model, or Span kind to LLM, and use Service or Agent to cover whole runs.

Because the Step filter matches exact span names, an alert written against an earlier SDK’s span names needs the new name after your agents move to the 1.0.0 SDKs. Those SDKs name unnamed tool, agent, retrieval and model spans after their operation and target, such as execute_tool search_orders.

Set it up

1. Open the custom alert builder and add a filter. Choose the Step field, the operator =, and the span name of the step’s LLM call. Pick it from the suggested values so the spelling matches exactly.
2. Add a rule on the Input tokens metric. Choose an aggregation. Use avg to catch a steady rise in prompt size, or p95 to catch the largest prompts. Choose a comparison operator and a threshold in tokens.
3. Set the shared window, anywhere from 5 minutes to 24 hours.
4. Read the plain-language preview. It states exactly when the alert fires, and it becomes the alert’s description on its details page.
5. Attach a channel (webhook, email or Slack) so the firing reaches someone, and set the Renotify policy to off or every N minutes.

The finished alert reads like this:

FILTERS   Step = chat gpt-4o                (an LLM span name)
RULES     p95(Input tokens) > 20000         (your threshold)
WINDOW    15 minutes

If you also need the alert to depend on something else, add a rule. Rules combine with AND only, so every rule must be true for the alert to fire. For either-or behavior, create a second alert. An alert takes up to 10 filters with up to 50 values each, and up to 10 rules.

The full builder reference is on Custom alerts. The overview of how alerts fire and notify is on Alerts.

What you get

• An alert that fires on input tokens for the one step you named, and ignores other steps in the same agent.
• A choice between average, extremes and percentiles, so you can watch typical prompt size or the heaviest prompts.
• A plain-language description of the alert that you can check before you save it.
• Firings that record the observed value of every rule against its threshold, so you can see which condition tripped.
• Notifications through webhook, email or Slack.

You can also add a Model, Service, Agent, Span kind, Operation, Session or Instance filter next to the Step filter. For example, combine Step with Model to watch one step only when it runs on one model.

When something else fits

• If you care about the size of each single model call rather than the per-trace figure, Context usage is measured per LLM call and shows how much of the model’s context window a call filled.
• If you want the step to stay under a cost limit instead of a token limit, use the Cost metric with a Step filter on that step’s LLM span name.
• If you only want a summary of tokens and cost over time, not a trigger, use the usage and cost view on Home.

Related