Skip to Content

How do I alert on number of LLM calls for one span kind?

In Rius, you create a custom alert with the LLM calls metric and the count aggregation, for example count(LLM calls) > 200. The metric already counts only spans of kind LLM, so a Span kind filter set to LLM leaves the count unchanged, and any other kind leaves nothing to count. Scope the alert with Service, Agent or Model instead. LLM calls is measured per call, not per trace, so the count is a number of calls in the window.

How it works

A custom alert is one or more rules, combined with AND, evaluated over one rolling window. Optional filters run first and decide which spans the rules see. The rule is then aggregated over only those matched spans. See Custom alerts for the full builder, and Alerts for how alerts fire.

The Span kind filter. Span kind is a filter field that matches a span’s kind: LLM, tool, chain, retriever and so on. Text filters offer =, ≠, in and not_in, and the builder suggests values from your workspace’s live data. Other fields let you narrow further: Service, Agent, Step (the exact span name), Model, Operation, Session and Instance.

What counts as an LLM call. The LLM calls metric is an event metric with two aggregations: count and ucount (distinct traces). It counts inference calls, the spans the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model. So a Span kind filter that selects only those spans leaves nothing for the metric to count. Span kind = LLM matches the metric’s own scope, so it changes nothing. Leave the span kind unfiltered and narrow by Model, Service or Agent.

Per-trace and per-call aggregation.

  • Numeric metrics such as Cost, tokens and Duration use per-trace semantics. avg(Cost) is the average cost per trace in the window, not the workspace total.
  • Time to first token, Context usage and LLM calls are the exception. They are measured per LLM call, so their aggregations run over calls, not traces.
  • With LLM calls, count is the number of calls in the window. ucount is the number of distinct traces that contain matching calls.

A note on Duration: when a Step, Model, Span kind or Operation filter is on, Duration runs per trace from the earliest matched span’s start to the latest matched span’s end, not across the whole run. If you add a Duration rule to the same alert, keep that in mind.

Set it up

  1. Open the custom alert builder and add a filter on Service or Agent for the agent you want to watch. Skip Span kind, because the metric already limits itself to LLM calls. Filters apply before aggregation.
  2. Add a rule: metric LLM calls, aggregation count, a comparison operator and a threshold. The operators are <, <=, >, >= and ==. Use ucount instead if you want to count the traces that made matching calls.
  3. Set the window, from 5 minutes to 24 hours. The window applies to every rule in the alert.
  4. Check the plain-language preview. It states exactly when the alert fires and becomes the alert’s description on its details page.
  5. Attach a channel (webhook, email or Slack) and pick a Renotify policy, off or every N minutes.

The rule reads like this, using your own service name and threshold:

FILTERS Service = checkout-agent RULES count(LLM calls) > 200 WINDOW 15 minutes

An alert takes up to 10 filters with up to 50 values each, up to 10 rules, and rules combine with AND only. For either-or behavior, create a second alert. A count(LLM calls) < 1 rule over a generous window can act as a quiet-agent check, because it fires when no matching calls arrive.

What you get

  • An alert that fires when the number of LLM calls from the spans you selected crosses your threshold in the rolling window.
  • A firing record that shows the observed value of each rule against its threshold, so you can see which condition tripped.
  • Scope you control: Service, Agent, Model, Step, Operation, Session or Instance, so a spike in calls from one agent or model does not get lost in workspace-wide totals.
  • A choice between counting calls with count and counting traces with ucount.
  • A custom alert can be edited, and unlike the pre-defined alerts it can be deleted. Deleting it removes its firing history after you confirm.

When something else fits

  • If you want to alert on model spend rather than call volume, use a rule on Cost and filter Model, or Span kind to LLM. A filter that selects only tool, chain or agent spans measures no cost.
  • If you need a per-trace average or percentile of something numeric, such as cost or tokens, use the numeric metrics. They aggregate per trace, while LLM calls aggregates per call.
  • If you only want a summary of calls and cost over time rather than a trigger, see Usage and cost on Home.

Still stuck? Get help

Last updated on