How do I alert on output tokens for a single model?

Rius alerts on output tokens for one model when you create a custom alert with a Model filter and a rule on the Output tokens metric. The filter keeps only the spans for that model, so the rule measures only those calls. For example, p95(Output tokens) > 2000 with Model = claude-haiku-4-5-20251001 fires when the heaviest 5% of traces in the window exceed 2,000 output tokens from that model.

How it works

A custom alert is one or more rules, combined with AND and evaluated over one shared rolling window. Filters run first and select which spans feed the rules. Aggregation happens afterwards.

• The Model filter matches gen_ai.request.model on each span. Tokens sit on model-call spans (LLM, embedding and reranker), and the filter keeps only the spans whose model matches. Spans from other models, and tool, chain or agent spans, do not count toward the rule.
• Output tokens is a numeric metric. The builder offers avg, min, max, p50, p95 and p99.
• Per-trace aggregation. Numeric metrics use per-trace semantics. avg(Output tokens) is the average per trace in the window, not the workspace total. This differs from Time to first token, Context usage and LLM calls, which are measured per LLM call.
• Matched spans only. Because the rule sees only the matched spans, a trace counts toward the metric through its calls to the filtered model, not through the rest of the run.
• Operators. Compare with <, <=, >, >= or ==.

The Model filter offers =, ≠, in and not_in. Use in to cover several models in one alert. Use not_in to alert on every model except the ones you list. The builder suggests values from your workspace’s live data.

Set it up

1. Open the custom alert builder from Alerts.
2. Add a filter on Model, choose =, and pick the model from the suggested values. The suggestions come from spans your workspace already received, so the name matches what your code sent.
3. Add a rule on Output tokens. Choose the aggregation that fits what you want to catch:
   • avg for a shift in typical response length.
   • p95 or p99 for a tail of unusually long outputs.
   • max for the single largest trace.
4. Set the comparison and the threshold in tokens.
5. Set the window. It can be anywhere from 5 minutes to 24 hours. A short window reacts fast, and a long one smooths out noise.
6. Read the plain-language preview, which becomes the alert’s description. Attach your notification channels and save.

The alert definition looks like this:

FILTERS   Model = claude-haiku-4-5-20251001
RULES     p95(Output tokens) > 2000
WINDOW    15 minutes

The threshold here is an example. Pick one from what your traces show for that model.

Add more rules to narrow the alert, since rules combine with AND. For example, add count(LLM calls) > 50 so the alert fires only when the model also has real volume in the window. If you want either-or behaviour, create a second alert, because there is no OR and no nesting.

What you get

• An alert scoped to one model, so a verbose model does not hide behind a quiet one, and a model you rarely call does not trigger noise for the others.
• Firings that record the observed value of every rule against its threshold, so you can see which condition tripped.
• A details page with the generated description, the editable rule, attached channels and firings.
• Room to grow: an alert takes up to 10 filters with up to 50 values each, and up to 10 rules. Add a Service or Agent filter next to the Model filter to watch one model inside one agent.
• A custom alert that you can edit or delete later. Deleting removes its firing history after you confirm.

The rule measures spans that match the filter and nothing else. A Model filter selects model calls, so you do not need a separate Span kind filter for this alert.

When something else fits

• If you want to watch output tokens across every model, leave the Model filter off. Filter on Service or Agent to measure whole runs.
• If you want a summary rather than a trigger, the same metrics appear as a summary on Home, described in Monitoring usage and cost, with no alert to set up.
• If you care about spend rather than length, use the Cost metric with the same Model filter. Cost covers only models with a known price.

Related