How do I alert on output tokens for one span kind?

Rius lets you alert on output tokens for one span kind with a custom alert that has a Span kind filter and a rule on the Output tokens metric. For example, filter Span kind to LLM and add the rule avg(Output tokens) > 2000. The filter narrows the spans that feed the rule before anything is aggregated, so the rule measures only the spans of that kind. Output tokens are aggregated per trace, not per call.

How it works

A custom alert is one or more rules, combined with AND, evaluated over one shared rolling window. Filters apply first. They select which spans feed the rules, and each filter matches a span’s own value.

• The filter this needs. Span kind is one of the filter fields, with values such as LLM, tool, chain and retriever. Text fields offer =, ≠, in and not_in, and values are suggested from your workspace’s live data.
• Why LLM is the usual choice. Tokens are recorded on model-call spans: LLM, embedding and reranker. A filter that selects only tool, chain or agent spans measures no tokens, so a rule on those kinds never has a value to compare. To alert on model output, set Span kind to LLM. If your agent also calls embedding or reranker models, use in and list those kinds too.
• Output tokens is a numeric metric. It offers avg, min, max, p50, p95 and p99, with comparison operators <, <=, >, >= and ==.
• How it aggregates. Numeric metrics use per-trace semantics. avg(Output tokens) is the average output tokens per trace in the window, not the workspace total, and with a Span kind filter each trace counts only its matched spans. Time to first token, Context usage and LLM calls are the exception, because they are measured per LLM call. Output tokens is not an exception, so its aggregations are over traces.
• Limits. An alert takes up to 10 filters with up to 50 values each, up to 10 rules, and a window between 5 minutes and 24 hours.

Set it up

1. Open the custom alert builder, as described on Custom alerts.
2. Add a filter: choose the Span kind field, pick =, and select LLM. To cover more model-call kinds, switch to in and add Embedding and Reranker.
3. Add a rule: choose the Output tokens metric, an aggregation such as avg or p95, a comparison operator and your own threshold. Pick p95 or max if you care about the few largest responses rather than the average.
4. Set the window, between 5 minutes and 24 hours. All rules in the alert share it.
5. Read the plain-language preview. It states exactly when the alert fires and becomes the alert’s description.
6. Attach a channel (webhook, email or Slack) and choose a Renotify policy, off or every N minutes.

The alert you build reads like this:

FILTERS   Span kind = LLM
RULES     avg(Output tokens) > 2000
WINDOW    15 minutes

The threshold of 2000 is only an example, so set it from what your own agents produce. To add a second condition, such as an error count, add another rule. Rules combine with AND only, so for either-or behavior create a second alert.

What you get

• An alert that watches output tokens on one span kind rather than on every span in the workspace.
• A value you can reason about: output tokens per trace, for the matched spans only, so avg, p95 or max describe typical or worst-case traces in the window.
• A details page with the generated description, the editable rule, attached channels and firings. Each firing records the observed value of every rule against its threshold, so you can see which condition tripped.
• The option to delete a custom alert later. Deleting it removes its firing history after an explicit confirmation.

For how firings, channels and renotification behave across all alerts, see Alerts.

When something else fits

• If you want to measure whole runs rather than the spans of one kind, filter by Service or Agent instead of Span kind.
• If you want to catch failure patterns rather than a token threshold, the pre-defined alerts cover patterns such as recurring errors, swallowed errors and tool loops without a rule to write.

Related