How do I alert on output tokens for a single agent?

Rius lets you alert on output tokens for one agent with a custom alert that has an Agent filter and a rule on the Output tokens metric. The filter limits the alert to spans from that agent, and the rule sets the threshold, such as p95(Output tokens) > 4000, over a rolling window you choose. Output tokens are aggregated per trace, so the rule compares a typical or worst-case run against your threshold, not a workspace total.

How it works

A custom alert is one or more rules combined with AND, all evaluated over one shared rolling window. Filters apply first and decide which spans feed the rules. Aggregation happens after that. For this question you need two pieces.

• The filter: set Agent equal to your agent’s name. Agent is the agent identity Rius derives, as each span carries it. The filter matches every span that carries that value, not only root spans, so the whole run of that agent is covered.
• The rule: pick Output tokens, an aggregation and a threshold. Output tokens is a numeric metric, and the builder offers avg, min, max, p50, p95 and p99 for it, in tokens.

How the aggregation reads. Numeric aggregations use per-trace semantics. avg(Output tokens) is the average output tokens per trace in the window, not the sum of all tokens the agent produced. p95(Output tokens) > 4000 fires when the heaviest 5% of traces in the window go over 4,000 output tokens.

A few metrics are the exception and are measured per LLM call instead of per trace: Time to first token, Context usage and LLM calls. Output tokens is not one of them, so you do not need to think in calls here.

Pick the filter that matches your goal. Service and Agent measure whole runs. Step, Model and Span kind filters narrow the matched spans, so the metric then covers only those spans. Use them only when you want output tokens for one step or one model, not for the agent as a whole.

Set it up

1. Open the custom alert builder in Alerts and start a new custom alert.
2. Add an Agent filter. Text fields offer =, ≠, in and not_in, and the values are suggested from your workspace’s live data, so you can pick your agent from the list. Use in to cover several agents with one alert.
3. Add a rule on Output tokens. Choose an aggregation and a comparison operator (<, <=, >, >= or ==), then enter the token threshold.
4. Set the window. It can be anywhere from 5 minutes to 24 hours and is shared by every rule in the alert.
5. Read the plain-language preview. It states exactly when the alert fires and becomes the alert’s description on its details page.
6. Attach a channel (webhook, email or Slack) and choose a Renotify policy, off or every N minutes.

The finished alert reads like this. The agent name and the threshold are examples, so replace them with your own:

FILTERS   Agent = support-agent
RULES     p95(Output tokens) > 4000
WINDOW    15 minutes

To require a second condition, add another rule. Rules combine with AND only, so the alert fires when both hold in the same window. For example, add count(Errors) > 10. There is no OR and no nesting. If you want either-or behaviour, create a second alert.

An alert takes up to 10 filters with up to 50 values each, up to 10 rules, and a window between 5 minutes and 24 hours. The full list of filters, metrics and aggregations is on Custom alerts.

What you get

• An alert that watches one agent, or a list of agents, and ignores the rest of your workspace.
• A firing that records the observed value of every rule against its threshold, so you can see which condition tripped.
• A generated description of the alert on its details page, with the editable rule, the attached channels and the firings.
• Notifications on the channels you attach, repeated on the schedule you set with Renotify.
• A custom alert you can edit at any time, and delete when you no longer need it. Deleting it removes its firing history after an explicit confirmation.

Pick the aggregation by what you want to catch:

• avg shows a general rise in how much each run writes.
• p95 or p99 catches the long tail of heavy runs while one-off runs stay quiet.
• max fires on a single run that goes over your limit.
• min fits a low-side check, such as min(Output tokens) falling under a value.

When something else fits

• If you want a total of output tokens across the workspace, an alert on this metric does not give it. The aggregation is per trace, and avg is not a sum. The same metrics are available as a summary on Home in the usage and cost view.
• If you want an alert on one step or one model instead of the whole agent, use a Step or Model filter in place of Agent, and expect the metric to cover only the matched spans.
• If you need either-or logic, for example two agents with different thresholds, create one alert per condition.

Related