How do I get an alert on my agent’s reasoning tokens?

Rius lets you alert on reasoning tokens with a custom alert that puts a threshold on the Reasoning tokens metric over a rolling window. Pick an aggregation (avg, min, max, p50, p95 or p99), set the comparison and a window between 5 minutes and 24 hours, and attach a channel. When the condition matches, Rius records a firing with the observed value and delivers it to every attached channel.

How it works

Reasoning tokens is a numeric metric in custom alerts. It is measured in tokens and offers six aggregations: avg, min, max, p50, p95 and p99.

The aggregation is taken per trace. avg(Reasoning tokens) is the average reasoning tokens per trace in the window, not the workspace total. Pick the aggregation by what you want to catch:

• avg catches a general shift, for example a prompt change that makes every run think longer.
• p95 or p99 catches the heaviest runs while ignoring normal ones. This is the usual choice for runaway reasoning.
• max fires on a single trace that crosses the line, so it is the most sensitive and the noisiest.
• min and p50 are available too, but they rarely fit a ceiling on spend.

An alert is one or more rules, always combined with AND, all evaluated over one shared window. Rius evaluates a rule every minute while it has no open firing, and every five minutes while one is open, so timings are approximate to that cadence.

A firing opens the first time the condition matches and stays open while it keeps matching. It resolves on its own after the condition has stopped matching for two hours. If the condition returns later, that is a new, unacknowledged firing.

Set it up

Creating an alert is an admin action. Members can see alerts and acknowledge firings.

1. Open the Alerts page and start a new custom alert. See Alerts for the page layout.
2. Optionally add filters, which are applied before aggregation. Filter by Service to watch one agent, by Model (gen_ai.request.model) to watch only your reasoning model, or by Step to watch one named span.
3. Add a rule on Reasoning tokens, choose the aggregation, the comparison and the threshold.
4. Set the window. A longer window, such as one hour, smooths out single odd runs. A short window, such as 15 minutes, reacts faster but fires more easily on a handful of traces.
5. Attach one or more distribution channels (webhook, email or Slack) and choose a Renotify policy: off, or every 5 to 1440 minutes until someone acknowledges.
6. Check the plain-language preview of when the alert fires, then save. That sentence becomes the alert’s description.

A rule that fits a typical case looks like this. The numbers are examples, so replace them with values taken from your own traces:

FILTERS   Service = checkout-agent
RULES     p95(Reasoning tokens) > 8000
   AND    count(Traces) > 20
WINDOW    1 hour

The second rule is optional. Because rules combine with AND, count(Traces) > 20 keeps the alert quiet in a window with so little traffic that a percentile means little. Choose the threshold from what your agent normally uses, set just above that level, then adjust after a few days of firings.

To cover both a drift and a spike, create two alerts, for example one on avg and one on p99. Rules never combine with OR.

What you get

• A firing for each occurrence. It records the observed value of every rule against its threshold, so you can see which condition tripped and by how much.
• A firing page. It shows when the firing opened, when the condition was last seen, and the affected traces, each linked to the trace view. It also lists one delivery row for each channel with the outcome of each attempt.
• Delivery to every attached channel. Each firing goes to all channels on the alert while it is enabled. With Renotify on, repeats appear as further rows in the delivery list until you acknowledge or the firing resolves.
• Acknowledgment. It records who saw the firing and stops further notifications for it. The firing still resolves by itself once the condition clears.
• Optional root-cause analysis. If you switch it on for the alert and your workspace has it enabled, each firing is handed to the analysis agent, and its report appears on the firing page.
• A sidebar count. The Alerts entry shows how many open firings nobody has acknowledged.

If you edit and save the alert, its open firings resolve and Rius evaluates the rule afresh. A condition that still holds opens a new firing under the new settings.

When something else fits

• If you want a fixed detector with no thresholds to tune, the pre-defined alerts for repeated errors, swallowed errors and tool loops cover those cases without a rule.
• If you want to watch reasoning tokens as a summary rather than a trigger, the usage and cost views on Home show tokens without notifying anyone.
• If you want either-or logic across several conditions, create one alert for each condition, since a single alert only combines rules with AND.

Related