How do I alert on context window usage for one span kind?

In Rius, you create a custom alert with a Span kind filter set to LLM and a rule on the Context usage metric, such as p95(Context usage) > 0.8. The filter is applied before aggregation, so the rule reads only the matched spans. Context usage is measured per LLM call, so the alert fires on the fullest calls in the window, not on whole traces.

How it works

A custom alert is one or more rules, combined with AND, evaluated over one shared rolling window. Filters pick which spans feed the rules, before any aggregation. Each filter matches a span’s own value.

Context usage is the share of the model’s context window that one LLM call filled. It is the call’s input tokens, prompt cache included, divided by the model’s window. You compare it against a value between 0 and 1, and the builder shows it as a percentage. The aggregations are avg, min, max, p50, p95 and p99.

Two details decide which span kind this works for:

• Only LLM calls carry context usage. An LLM call is an inference call, a span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model. A Span kind filter on tool, chain or agent measures no context usage.
• The aggregation is over calls, not traces. Most numeric metrics use per-trace semantics, so avg(Cost) is the average cost per trace. Context usage, Time to first token and LLM calls are the exception. With p95(Context usage) > 0.8, the 95th percentile is taken across LLM calls in the window, and the alert fires when the fullest 5% of calls use more than 80% of their model’s context.

A call whose model has no known context window has no context usage. Rius leaves it out and never counts it as 0.

Set it up

1. Open the custom alert builder, as described on Custom alerts.
2. Add a filter: choose the Span kind field and set it to LLM. Text fields offer =, ≠, in and not_in, and values are suggested from your workspace’s live data.
3. Optionally narrow the scope further with a Service, Agent, Model or Step filter. A Step filter matches span names exactly.
4. Add a rule on Context usage, pick an aggregation such as p95, and set a comparison and a value between 0 and 1. The comparison operators are <, <=, >, >= and ==.
5. Set the window. It can be anywhere from 5 minutes to 24 hours, and all rules share it.
6. Attach a channel and save. The builder’s plain-language preview becomes the alert’s description.

The alert looks like this:

FILTERS   Span kind = LLM
RULES     p95(Context usage) > 0.8
WINDOW    15 minutes

To watch several span kinds that carry model calls, use in on the Span kind filter. Only the LLM calls among them contribute context usage.

If you add a second rule, it must also hold for the alert to fire, because rules combine with AND only. An alert takes up to 10 filters with up to 50 values each, and up to 10 rules.

What you get

• An alert that fires when LLM calls fill too much of their model’s context window, scoped to the calls you chose.
• A reading that reflects individual calls. One nearly full call is not averaged away by the other steps in a trace.
• Firings that record the observed value of every rule against its threshold, so you can see which condition tripped.
• A coverage ratio on each firing, so you can tell a real reading from one based on few calls with a known context window. Coverage is attached only to firings. A rule held under its threshold by low coverage stays quiet, so check that your models have a known context window before you rely on the alert.

Keep in mind that a filter changes what every metric in the alert covers. If you add a Duration rule next to the context rule, Duration runs, per trace, from the earliest matched span’s start to the latest matched span’s end, not across the whole run. For whole-run measurements, use a Service or Agent filter in a separate alert.

When something else fits

• If you want to watch whole runs, such as total time per trace, use a Service or Agent filter instead of a Span kind filter.
• If you want either-or conditions, create a second alert, because rules combine with AND only.
• For an overview of how alerts fire, notify and renotify, see Alerts.

Related