How do I alert on share of failed LLM calls for a single agent?
Rius lets you do this with a custom alert on the LLM error rate metric, filtered to Agent equals your agent’s name. LLM error rate is the share of LLM calls that errored in the rolling window, and you compare it against a value between 0 and 1. The Agent filter limits the rule to spans that carry that agent, so other agents in the workspace do not affect it.
How it works
A custom alert is one or more rules, combined with AND, over one shared rolling window. Filters apply first and select which spans feed the rules. Aggregation comes after that.
- The metric. LLM error rate is a ratio metric. Its only aggregation is
rate, because a rate is already an aggregate. It is the share of LLM calls that errored in the window, as a value from 0 to 1. An LLM call is an inference call, the span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model. - The filter. Agent matches the agent identity Rius derives, as each span carries it. It selects every span that carries the value, not only root spans, so the model-call spans inside the agent’s runs are included.
- Per-trace and per-call metrics. Numeric metrics such as Cost, Duration and the token counts use per-trace semantics.
avg(Cost)is the average cost per trace in the window, not the workspace total. Time to first token, Context usage and LLM calls are measured per LLM call, so their aggregations run over calls. Error rate is the share of spans that errored, and LLM error rate is the share of LLM calls that errored. - Other filters change coverage. Step, Model, Span kind and Operation filters change which spans a metric covers. For this alert you do not need them. Use Agent, or Service to cover everything one service sends.
Set it up
- Open the custom alert builder and add a filter: Agent, operator
=, and your agent’s name. Values are suggested from your workspace’s live data, so you can pick the name from the list. Useinif the same agent goes by more than one name. - Add a rule with the metric LLM error rate, the aggregation
rate, a comparison operator and a value between 0 and 1. For example,>and0.05fires when more than 5% of the agent’s LLM calls errored. The operators are<,<=,>,>=and==. - Set the window, from 5 minutes to 24 hours. All rules in the alert share it.
- Read the plain-language preview in the builder. It states exactly when the alert fires, and it becomes the alert’s description on its details page.
- Attach the channels that should receive firings, and set the Renotify policy to off or every N minutes.
The finished alert reads like this in the rule grammar:
FILTERS Agent = your-agent-name
RULES LLM error rate > 0.05
WINDOW 15 minutesTo require a minimum amount of traffic before the alert fires, add a second rule, such as a count on LLM calls. Rules combine with AND, so both must hold in the same window.
What you get
- An alert scoped to one agent. Its failed-call share is measured only over that agent’s LLM calls, so a noisy neighbour in the same workspace does not trigger it.
- A ratio you can read directly. Because the metric is already a share of calls, you set a threshold like 0.05 rather than working out a count from call volume.
- A details page with the generated description, the editable rule, the attached channels and the firings. Each firing records the observed value of every rule against its threshold, so you can see which condition tripped.
- A rule you can change at any time, and an alert you can delete when you no longer need it. Deleting removes its firing history after an explicit confirmation.
For the wider picture of how alerts fire, renotify and reach your channels, see Alerts.
When something else fits
- If you want the share of all spans that errored, tool and chain steps included, use the Error rate metric instead. If you want a count rather than a share, use
count(Errors). - If you want either-or logic, such as one threshold on error share or another on cost, create a second alert. Rules inside one alert combine with AND only.
- If you want to cover every agent in a service, filter on Service instead of Agent.