How do I alert on number of LLM calls for one step?
In Rius, create a custom alert with a Step filter set to the span name and a rule of count on the LLM calls metric, then choose a threshold and a window. The alert fires when the model-call spans with that name exceed your threshold inside the window. The Step filter matches the span’s own name exactly, so use the name of the span that makes the model call.
How it works
A custom alert is one or more rules, combined with AND, evaluated over one rolling window. Filters apply first and select which spans feed the rules. Aggregation comes after.
- The filter this needs. Step matches the span name exactly. Each filter matches a span’s own value, so Step = your step name selects the spans carrying that name.
- What counts as an LLM call. An LLM call is an inference call, a span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model.
- How the metric aggregates. LLM calls is an event metric with two aggregations.
countis the number of matching calls in the window.ucountis the number of distinct traces that contain a matching call. - Per call, not per trace. Cost, tokens and Duration are averaged per trace, so
avg(Cost)is the average cost per trace, not the workspace total. LLM calls, Time to first token and Context usage are measured per LLM call, so their aggregations run over calls. - Where the Step filter stops. Because rules measure only matched spans, a Step filter that names a chain, agent or tool span matches no LLM calls. Filtering on a parent step does not count the model calls nested inside it. Filter on the name of the model-call span itself.
- Narrowing further. Add a Model or Service filter next to Step. Text filters offer
=,≠,inandnot_in.
Set it up
- In the custom alert builder, add a Step filter. Pick the span name of the model call from the suggested values, which come from your workspace’s live data. Use
into cover several names. - Optionally add a Service, Agent or Model filter to narrow the match.
- Add one rule: metric LLM calls, aggregation
count, operator>, and your threshold. Useucountif you want to count traces, not calls. - Set the window, anywhere from 5 minutes to 24 hours. All rules share it.
- Check the plain-language preview. It becomes the alert’s description on its details page.
- Attach channels and set Renotify to off or every N minutes.
The finished alert reads like this. The step name is a placeholder for your own span name.
FILTERS Step = your-step-name
RULES count(LLM calls) > 50
WINDOW 15 minutesComparison operators are <, <=, >, >= and ==. An alert takes up to 10 filters with up to 50 values each, and up to 10 rules.
The Step filter needs the exact name. The 1.0.0 SDKs name unnamed tool, agent, retrieval and model spans after their operation and target, such as execute_tool search_orders. An alert written against spans from an earlier SDK needs the new name once your agents upgrade. The full filter and metric reference is on Custom alerts. For how alerts fire and are routed, see Alerts.
What you get
- An alert scoped to the model-call spans with one name, so a runaway loop around a single step is visible without noise from the rest of the run.
- A count of calls with
count, or a count of affected traces withucount. - Firings that record the observed value of every rule against its threshold, so you see which condition tripped.
- A generated description of when the alert fires, taken from the builder preview.
- A details page with the editable rule, attached channels and firing history. A custom alert can be deleted, which removes its firing history after an explicit confirmation.
When something else fits
- To count all LLM calls in a run, drop the Step filter and filter by Service or Agent instead. Those measure whole runs.
- For either-or conditions, create a second alert. Rules combine with AND only, with no OR and no nesting.
- If the model-call span name varies, use
inwith several names, or filter by Model instead of Step. A Span kind filter set to LLM adds nothing here, because the metric already counts only LLM calls.