How do I get an alert on my agent’s context window usage?

Rius alerts on context window usage with a custom alert on the Context usage metric, for example p95(Context usage) > 0.8 over a 15 minute window. Context usage is the share of the model’s context window that one LLM call filled, so the alert fires when your fullest calls approach the limit. Each firing records the observed value against your threshold and goes to every channel attached to the alert.

How it works

A custom alert is one or more rules, combined with AND, evaluated over one rolling window. Context usage is one of the metrics you can put a rule on.

• What it measures. For each LLM call, the input tokens (prompt cache included) divided by the model’s context window. You compare it against a value between 0 and 1, and the builder shows it as a percentage.
• Per call, not per trace. Unlike cost, the aggregation runs over LLM calls. An LLM call is an inference call, a span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model.
• Aggregations. avg, min, max, p50, p95 and p99.
• Models without a known window. A call whose model has no known context window has no context usage. It is left out, never counted as 0.
• Evaluation. Rules are evaluated every minute while the alert has no open firing, and every five minutes while one is open.

Set it up

Creating an alert is an admin action. Open the Alerts page, add a custom alert, and build the rule.

1. Pick the aggregation. Use p95 to catch the fullest 5% of calls, which is the usual choice for a limit you do not want to hit. Use max if a single call near the limit matters. Use avg only to watch a general drift, because it hides a few very full calls.
2. Set the threshold. Enter a value between 0 and 1 with the > operator. 0.8 means calls that fill more than 80% of their model’s context. Lower it if you want earlier warning, raise it if the alert is noisy.
3. Set the window. The window runs from 5 minutes to 24 hours. Choose one long enough to hold a meaningful number of LLM calls. 15 minutes suits an agent with steady traffic, and a few hours suits a low-traffic agent, because a percentile over only a handful of calls reflects little.
4. Scope it with filters (optional). Filter by Service, Agent, Step, Model, Span kind, Operation, Session or Instance. A filter on Model is useful here, since each model has its own window. Filters apply before aggregation.
5. Attach channels and set Renotify. Choose where firings go (webhook, email or Slack) and whether to repeat the notification every 5 to 1440 minutes until someone acknowledges it.

The rule reads like this in the builder:

FILTERS   Service = checkout-agent
RULES     p95(Context usage) > 0.8
WINDOW    15 minutes

The builder turns the rule into a plain-language sentence that becomes the alert’s description on its details page. Check it before you save. Saving any later change resolves the alert’s open firings and starts evaluating the rule afresh.

What you get

• A firing when calls run near the limit. It opens the first time the condition matches and stays open while it keeps matching.
• The observed value next to your threshold. Each firing records the observed value of every rule against its threshold, so you can see which condition tripped.
• A coverage ratio. Firings on Context usage show how many calls had a known context window, so you can tell a real reading from a partly covered one. Coverage is attached only to firings. A rule held under its threshold by low coverage stays quiet.
• Delivery to every attached channel. The firing’s page lists one row per channel with the outcome of each attempt.
• Controlled noise. An acknowledged firing stops further notifications. The firing resolves on its own once the condition has stopped matching for two hours, and a return of the condition opens a new, unacknowledged firing.
• Optional root-cause analysis. Turn on the alert’s Root-cause analysis switch to have each firing analyzed. See Alerts for firings, acknowledgment and analysis status.

When something else fits

If you want to see context usage rather than be notified, open the agent on the Agents page: its Context section shows Peak usage, the share of the window the largest call used, and what filled the prompt. If one alert must fire on either of two conditions, create a second alert, because rules combine with AND only.

Related