How do I get an alert on my agent’s input tokens?

Rius lets you build a custom alert on the Input tokens metric, so you are notified when your agent’s prompts grow past a threshold you set. Pick an aggregation (avg, min, max, p50, p95 or p99), a threshold in tokens and a rolling window of 5 minutes to 24 hours, then attach a channel. For most agents, p95 over an hour catches the heaviest runs without firing on one outlier.

How it works

A custom alert is one or more rules that you combine with AND, all evaluated over one shared rolling window. A rule on input tokens has the form aggregation, comparison, threshold, for example p95(Input tokens) > 20000. The comparison operators are <, <=, >, >= and ==.

Input tokens is a numeric metric measured per trace. avg(Input tokens) is the average input tokens per trace in the window, not the workspace total. Choose the aggregation by what you want to catch:

• avg: a slow drift upward, such as a prompt that keeps growing.
• p95 or p99: the heaviest few percent of traces, which is where runaway context shows up first.
• max: any single trace over the line. This is the noisiest choice.
• min: a floor, for example to notice that a prompt has been cut off. Use it with <.

Use optional filters to limit which spans feed the rule before it aggregates. You can filter by Service, Agent, Step, Model, Span kind, Operation, Session or Instance. A Service or Model filter keeps one chatty agent from hiding another.

Rius checks a rule every minute while it has no open firing, and every five minutes while one is open, so timing is approximate to that cadence. A firing stays open while the condition keeps matching and resolves on its own after two hours without a match. If the condition returns later, that is a new firing.

Set it up

1. Open the Alerts page and go to the Custom alerts section. Creating an alert is an admin action. Members can see alerts and acknowledge firings.
2. Add a filter if you want one, such as your agent’s service name, so the rule watches only that agent.
3. Add a rule on Input tokens. Start with p95 and a threshold a little above what a normal trace uses today. Check typical values first, such as the Tokens column of the trace list, then adjust the example below to your own numbers.
4. Set the window. A window of 30 to 60 minutes smooths out single heavy runs. A short window of 5 to 15 minutes reacts faster but fires more often. The limits are 5 minutes to 24 hours.
5. Attach one or more distribution channels (webhook, email or Slack) and choose a Renotify policy. Off sends one notification per firing. Every N minutes, from 5 to 1440, repeats it until someone acknowledges the firing or it resolves.
6. Save. The builder shows a plain-language preview of when the alert fires, and that sentence becomes the alert’s description.

An example rule, with the service name and numbers replaced by yours:

FILTERS   Service = checkout-agent
RULES     p95(Input tokens) > 20000
WINDOW    1 hour

If you edit the alert later, Rius resolves its open firings and starts evaluating afresh. A condition that still holds opens a new firing under the new settings.

What you get

When the rule matches, Rius opens a firing and delivers it to every channel attached to the alert. The firing records the observed value of each rule against its threshold, so you can see how far over the line the p95 or average went.

The firing’s own page shows when it opened, when the condition was last seen, the affected traces linked to the trace view, and a delivery list with one row per channel and the outcome of each attempt. View all in Traces opens the trace list filtered to the firing’s time window, so you can go straight to the prompts that grew.

Acknowledging a firing records who saw it and stops further notifications for it. The firing still resolves by itself once the condition clears. The Alerts entry in the sidebar shows the count of open firings nobody has acknowledged. If you add a second rule to the same alert, for example on Errors, both must hold in the window before it fires.

When something else fits

• If what you care about is how full the model’s context window is, rather than a raw token count, use the Context usage metric, for example p95(Context usage) > 0.8. It is measured per model call and compares against the model’s own window.
• If you want a total or a view of spend rather than a per-trace figure, read it on Usage and cost. The Cost alert metric is per trace too, and a total-spend threshold is not an alert metric.

Related