Skip to Content

How do I alert on number of LLM calls for a single model?

Rius lets you create a custom alert with the rule count(LLM calls) and a Model filter set to the model you care about. The filter limits the rule to spans for that model before anything is counted, so the alert fires when that model’s call count crosses your threshold inside the window. Only inference calls are counted, so embedding, tool, agent and retrieval spans never add to the number.

How it works

A custom alert is one or more rules combined with AND, all evaluated over one shared rolling window. Optional filters run first and decide which spans feed the rules. Aggregation happens after that.

Two parts of the alert do the work for this question:

  • The Model filter matches gen_ai.request.model. Text filters offer =, ≠, in and not_in, and the builder suggests values from your workspace’s live data. For a single model, use =.
  • The LLM calls metric is an event metric. Its aggregations are count and ucount. count totals the calls in the window. ucount counts distinct traces that contain at least one matching call.

An LLM call is an inference call, a span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model. That means a Model filter on its own already gives you the calls to that model and nothing else.

This metric also aggregates differently from the numeric ones. Numeric metrics such as Cost, Duration and the token metrics use per-trace aggregation, so avg(Cost) is the average cost per trace in the window, not a workspace total. Time to first token, Context usage and LLM calls are the exception. They are measured per LLM call, so their aggregations run over calls, not traces. For your rule, count(LLM calls) is a number of calls, and ucount(LLM calls) is a number of traces.

Set it up

  1. Open the custom alert builder and add a filter: Model, operator =, and the model name picked from the suggested values.
  2. Add one rule: metric LLM calls, aggregation count, comparison > (the operators are <, <=, >, >= and ==), and your threshold. Choose ucount instead if you want to count traces that used the model rather than individual calls.
  3. Set the window. It can be anywhere from 5 minutes to 24 hours. The count is taken over that rolling window, so pick a threshold that matches its length.
  4. Check the plain-language preview. It states exactly when the alert fires, and that sentence becomes the alert’s description on its details page.
  5. Attach your distribution channels and save.

The rule reads like this:

FILTERS Model = claude-haiku-4-5-20251001 RULES count(LLM calls) > 500 WINDOW 1 hour

Swap in your own model name and threshold. If you want the alert to fire when calls to the model stop, use < with a low threshold over a window long enough to cover normal gaps. Only add more rules if you want them all to hold at once, because rules combine with AND and there is no OR. For either-or behavior, create a second alert.

The full builder reference is on Custom alerts.

What you get

  • An alert scoped to one model’s inference calls, with a count that no tool, agent, retrieval or embedding span can inflate.
  • A firing record that shows the observed value of the rule against its threshold, so you can see which condition tripped.
  • A generated description of the alert on its details page, taken from the builder’s preview.
  • A rule you can edit later, and an alert you can delete. Deleting it removes its firing history after you confirm.

An alert takes up to 10 filters with up to 50 values each, and up to 10 rules. See Alerts for how firings, channels and renotify policies fit together.

When something else fits

  • To watch several models together, use in on the Model filter, or drop the Model filter to count every inference call, since the metric already counts only LLM calls.
  • To track spend on that model rather than volume, use the Cost metric with the same Model filter. It is a per-trace number, so avg(Cost) is average cost per trace.
  • To watch whole runs rather than one model’s calls, filter by Service or Agent instead of Model.

Still stuck? Get help

Last updated on