How do I alert on share of failed LLM calls for one service?
Rius lets you alert on the share of failed LLM calls for one service with a custom alert that uses the LLM error rate metric and a Service filter. The metric is the share of LLM calls that errored in the rolling window, and you compare it against a value between 0 and 1. For example, LLM error rate > 0.05 with Service = checkout-agent fires when more than 5% of that service’s LLM calls fail.
How it works
A custom alert is one or more rules, combined with AND, evaluated over one shared rolling window. Filters run first. They select which spans feed the rules, before any aggregation, and each filter matches a span’s own value.
For this question, two pieces matter:
- The filter. Set Service to the
service.nameyour instrumentation sends. Rius suggests values from your workspace’s live data. Useinif you want several services in one alert, ornot_into exclude some. - The metric. LLM error rate is a ratio metric. Its only aggregation is
rate, because a rate is already an aggregate. You do not choose avg or p95 for it.
An LLM call is an inference call, the span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model. So a failing tool does not raise this metric. That is the difference from Error rate, which is the share of all spans that errored.
How other metrics aggregate matters when you add more rules:
- Numeric metrics such as Cost, Duration and the token metrics use per-trace semantics.
avg(Cost)is the average cost per trace in the window, not the workspace total. - Time to first token, Context usage and LLM calls are measured per LLM call, so their aggregations run over calls, not traces.
- Event metrics such as Errors, Traces and LLM calls use
count, orucountfor distinct traces.
Keep the filters narrow in meaning. Step, Model, Span kind and Operation filters change what a metric covers, because the rules measure only the matched spans. A Model filter, for example, limits the rate to calls to that model. Use Service alone when you want the failure share across every LLM call the service makes.
Set it up
- Open the custom alert builder from Alerts and create a custom alert. The full builder is described on Custom alerts.
- Add a filter: field Service, operator
=, value your service name, for examplecheckout-agent. - Add a rule: metric LLM error rate, aggregation
rate, operator>, value0.05. Enter the threshold as a fraction, so 5% is0.05and 20% is0.2. - Set the window. It can be between 5 minutes and 24 hours, and all rules share it. A longer window smooths out a few failures in a quiet period. A shorter window reacts faster.
- Optionally add a second rule so a handful of calls cannot trip the alert. Rules combine with AND, so both must hold.
- Read the plain-language preview. It states exactly when the alert fires, and it becomes the alert’s description on its details page.
- Attach a channel and set the Renotify policy, then save.
The alert then reads like this:
FILTERS Service = checkout-agent
RULES rate(LLM error rate) > 0.05
AND count(LLM calls) > 20
WINDOW 15 minutesThe count(LLM calls) rule is optional. It keeps the alert quiet when the service made only a few calls in the window, where one failure would look like a large share. An alert can hold up to 10 filters with up to 50 values each, and up to 10 rules. If you want either-or logic, such as a high failure share or a high cost, create a second alert, because there is no OR.
What you get
- An alert scoped to one service that watches the failure share of its model calls, not of every span.
- A threshold you read as a percentage: 0.05 is 5% of LLM calls in the window.
- A details page with the generated description, the editable rule, the attached channels and the firings. Each firing records the observed value of every rule against its threshold, so you can see which condition tripped.
- The option to guard the ratio with a call count, so a low-traffic window does not produce a noisy firing.
- Edit and delete control. Unlike pre-defined alerts, a custom alert can be deleted. Deleting it removes its firing history after an explicit confirmation.
When something else fits
- If you want failures across all spans in the service, including tools and chains, use the Error rate metric with the same Service filter.
- If you want a raw count of failures instead of a share, use
count(Errors)with the Service filter. - If you do not want to build a rule, the pre-defined alerts cover common error patterns without a threshold to tune. See Pre-defined alerts.