How do I alert on reasoning tokens for a single agent?

Rius lets you alert on reasoning tokens for one agent with a custom alert that filters Agent to that agent and sets a rule on the Reasoning tokens metric. The filter picks which spans feed the rule, and the rule puts a threshold on an aggregation such as average or p95. Reasoning tokens are measured per trace, so the alert compares an aggregation of per-trace reasoning tokens, such as the average or p95 over the window, against your threshold.

How it works

A custom alert is one or more rules, combined with AND, over one shared rolling window. Optional filters run first and decide which spans the rules see.

• The filter. Agent matches the agent identity Rius derives, as each span carries it. It selects every span that carries that value, not only root spans, so the model-call spans inside the agent are included. Those spans hold the token counts.
• The metric. Reasoning tokens is a numeric metric. It offers avg, min, max, p50, p95 and p99, in tokens.
• Per-trace aggregation. Numeric aggregations use per-trace semantics. avg(Reasoning tokens) is the average per trace in the window, not the total for the agent. A few heavy traces raise a p95 or max rule even when the average stays low.
• Per-call metrics. Time to first token, Context usage and LLM calls work differently. They are measured per LLM call, so their aggregations run over calls, not traces. If you combine one of them with Reasoning tokens in the same alert, the two rules aggregate over different units.
• Rules and window. Rules combine with AND only, with no OR. For either-or behavior, create a second alert. The window is shared by all rules and runs from 5 minutes to 24 hours. An alert takes up to 10 filters with up to 50 values each, and up to 10 rules.

The builder shows a plain-language preview of when the alert fires, and that sentence becomes the alert’s description.

Set it up

1. Open the custom alert builder from Alerts and create a new custom alert.
2. Add a filter on Agent. Use = for one agent, or in for several. Values are suggested from your workspace’s live data, so pick the name from the list.
3. Add a rule on Reasoning tokens. Choose an aggregation and a comparison. Use avg for a sustained shift and p95 or max to catch heavy traces. The comparison operators are less than, less than or equal, greater than, greater than or equal, and equal.
4. Set the window. Choose one long enough to hold several traces, because the aggregation is over traces in that window.
5. Check the preview sentence, attach a channel (webhook, email or Slack) and save.

The alert reads like this:

FILTERS   Agent = my-agent
RULES     avg(Reasoning tokens) > 20000
WINDOW    15 minutes

Replace my-agent with your agent’s name and 20000 with your own threshold.

Do not scope this alert with a Span kind or Step filter that selects only tool, chain or agent spans. Tokens sit on model-call spans, so such a filter measures no reasoning tokens and the alert never fires. To measure whole runs, use Agent or Service. If you use a Span kind filter, set it to LLM.

What you get

• An alert limited to one agent, so other agents in the same service do not move the number.
• A reading of reasoning tokens per trace, which you can tune with avg for trends or p95 and max for outliers.
• A firing record that shows the observed value of every rule against its threshold, so you can see which condition tripped.
• Notifications through the channels you attach, and a custom alert you can edit or delete at any time. Deleting it removes its firing history after you confirm.
• A dead-man switch option: add a second alert with Traces count < 1 over a generous window to learn when ingestion stops.

When something else fits

• To cover every agent in a service, filter Service instead of Agent. To narrow to one process, add an Instance filter. To narrow to one model, add a Model filter.
• To alert on how full the context window is or on time to first token, use those metrics directly. They are per-call, so they aggregate over calls, not traces.
• If you want a ready-made check with no setup, the pre-defined alerts cover error patterns, swallowed errors, tool loops and context rot. They do not alert on reasoning tokens.

Related