How do I get an alert on my agent’s output tokens?

Rius lets you build a custom alert on the Output tokens metric, with a threshold on avg, min, max, p50, p95 or p99 over a rolling window of 5 minutes to 24 hours. Pick p95 or max to catch runaway replies and avg to catch a general drift. When the condition matches, Rius opens a firing that records the observed value against your threshold and delivers it to the channels attached to the alert. This is one of the rules you can build under Custom alerts.

How it works

Output tokens is a numeric metric, measured per trace. avg(Output tokens) is the average output tokens per trace in the window, not the workspace total. Rius evaluates the rule over a rolling window and compares the result with your threshold using <, <=, >, >= or ==.

Choose the aggregation by what you want to catch:

• p95 or p99: the heaviest traces. p95(Output tokens) fires when the top 5% of traces in the window are above your threshold. This suits a model that sometimes rambles or loops on long replies.
• max: a single trace that crosses the line. It is the most sensitive and the noisiest.
• avg or p50: a shift in the typical trace, for example after a prompt change makes every answer longer.
• min: a floor, for example to notice replies that suddenly shrink.

A rule can be narrowed with filters that apply before aggregation: Service, Agent, Step, Model, Span kind, Operation, Session and Instance. A Model filter limits the rule to one gen_ai.request.model, which helps when models have very different reply lengths. You can combine up to 10 rules in one alert, always with AND, and all of them share one window. For either-or behavior, create a second alert.

Rules are checked every minute while the alert has no open firing, and every five minutes while one is open, so timing is approximate to that cadence.

Set it up

Creating and editing alerts is an admin action. Members can see firings and acknowledge them.

1. Open the Alerts page and start a rule in the Custom alerts section. The builder previews in plain language exactly when the alert will fire, and that sentence becomes the alert’s description.
2. Add filters if you want a narrower scope, for example Service = checkout-agent.
3. Add a rule on Output tokens, choose an aggregation and a comparison, and enter a threshold in tokens. Base the number on what your agent normally produces, and set it clearly above that.
4. Set the window, between 5 minutes and 24 hours. A short window of 5 to 15 minutes reacts fast and suits p95 or max on a busy agent. An hour or more smooths out noise and suits avg on a quieter agent. A percentile over very few traces in the window swings widely, so lengthen the window if the alert flaps.
5. Set Renotify: off sends one notification per firing, or choose every N minutes, from 5 to 1440, to repeat until someone acknowledges it.
6. Attach one or more distribution channels (webhook, email or Slack) and save.

The finished rule reads like this. The service name and the numbers are examples, so use your own:

FILTERS   Service = checkout-agent          (optional, applied before aggregation)
RULES     p95(Output tokens) > 2000
WINDOW    15 minutes                        (shared by all rules)

For how alerts, firings and channels fit together, see Alerts.

What you get

• A firing for each occurrence. It opens the first time the condition matches and stays open while it keeps matching. It resolves on its own once the condition has stopped matching for two hours. A condition that returns later is a new, unacknowledged firing.
• The observed value against the threshold. The firing records the value of every rule next to its threshold, so you can see which condition tripped when an alert has several rules.
• A firing page. It shows when the firing opened, when the condition was last seen, the affected traces linked to the trace view, and a delivery list with one row per channel and the outcome of each attempt. Each renotify repeat adds a row.
• Delivery to every attached channel while the alert is enabled. Acknowledging a firing records who saw it and stops further notifications for it.
• Optional root-cause analysis. If you turn on the alert’s Root-cause analysis switch and the workspace has it enabled, each firing is handed to the analysis agent and its report appears on the firing page.

Saving a change to the alert resolves its open firings and restarts evaluation, so a condition that still holds opens a new firing under the new settings.

When something else fits

• To watch spend rather than reply length, build the same kind of rule on Cost, or on Input tokens or Reasoning tokens if those are the numbers you care about.
• For recurring errors, swallowed errors or loops, the pre-defined alerts need no rule at all, and you can turn each on or off.
• If you only want to look at token trends rather than be notified, the usage and cost summary shows the same metrics without a trigger.

Related