How do I alert on number of LLM calls for one service?
Rius lets you alert on the number of LLM calls for one service with a custom alert that has a Service filter and the rule count(LLM calls) > N. The filter limits the alert to spans from that service before anything is counted. LLM calls is measured per call, so the count is the number of inference calls in the rolling window.
How it works
A custom alert is one or more rules, combined with AND, evaluated over one shared rolling window. Filters apply first, then the rules aggregate what is left. See Custom alerts for the full grammar, and Alerts for how alerts fire.
- The filter this needs. Use the Service filter. It matches
service.namefrom your instrumentation, and the builder suggests values from your workspace’s live data. Use=for one service.inaccepts several services and counts their calls together. - The metric. LLM calls is an event metric. Its aggregations are
countanducount(distinct traces). - What counts as a call. An LLM call is an inference call, a span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model. You do not need a Span kind filter to keep tool or agent spans out of this count.
- Per trace and per call. Cost, token counts and Duration use per-trace semantics, so
avg(Cost)is the average cost per trace in the window, not a workspace total. Time to first token, Context usage and LLM calls are the exception. They are measured per LLM call, so their aggregations run over calls, not traces. For LLM calls,countgives the number of calls anducountgives the number of distinct traces. - Comparison and limits. The operators are
<,<=,>,>=and==. The window runs from 5 minutes to 24 hours. An alert takes up to 10 filters and up to 10 rules.
A Step, Model, Span kind or Operation filter narrows what a metric covers, because the rules measure only the matched spans. For a plain count of a service’s LLM calls, the Service filter is the one you need.
Set it up
- Open the custom alert builder and add a filter: set Service to
=and pick your service from the suggested values. To cover several services in one alert, chooseinand list them. - Add a rule: choose the LLM calls metric, the
countaggregation, a comparison operator and your threshold. Useucountinstead if you want the number of distinct traces that made calls. - Set the window. Pick a length between 5 minutes and 24 hours. All rules in the alert share it, so choose one that matches how fast you want to hear about a spike.
- Check the plain-language preview. The builder shows exactly when the alert will fire, and that sentence becomes the alert’s description on its details page.
- Attach a channel and, if you want repeat notifications, set the Renotify policy to off or every N minutes.
The alert reads like this. The service name and threshold are examples, so use your own:
FILTERS Service = checkout-agent
RULES count(LLM calls) > 500
WINDOW 15 minutesThis fires when checkout-agent makes more than 500 LLM calls within the rolling 15 minutes. To watch several services separately, create one alert per service, since rules combine with AND only and there is no OR.
What you get
- An alert scoped to one service that fires when its LLM call volume crosses your threshold in the window.
- A count of real inference calls only, with tool, agent, retrieval and embedding spans left out.
- A details page with the generated description, the editable rule, attached channels and firings. Each firing records the observed value of the rule against its threshold, so you can see what tripped it.
- The choice between calls (
count) and distinct traces (ucount) from the same metric. - A custom alert you can edit or delete. Deleting it removes its firing history after you confirm.
When something else fits
- If you care about spend rather than volume, filter Model, or Span kind to LLM, and alert on Cost. Cost is only on model-call spans, and it is only as complete as Rius’s pricing coverage.
- If you want a whole-run view such as Duration across a run, scope with Service or Agent rather than Step, Model, Span kind or Operation filters.
- If you want to be told when ingestion stops entirely, the Traces metric with a generous window works as a dead-man switch, as described in Custom alerts.