How do I alert on share of failed LLM calls for one span kind?
Rius has a built-in LLM error rate metric, the share of LLM calls that errored in the window, so you create a custom alert with a rule such as rate(LLM error rate) > 0.05. That metric already limits itself to LLM calls, so it needs no Span kind filter. For the share of failed spans of one kind, such as tools, use the Error rate metric with a Span kind filter instead.
How it works
A custom alert is one or more rules, combined with AND, evaluated over one shared rolling window. Filters apply first and select which spans feed the rules. Each filter matches a span’s own value.
Two ratio metrics cover failure share:
- LLM error rate is the share of LLM calls that errored in the window. An LLM call is an inference call, a span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model.
- Error rate is the share of spans that errored in the window.
Both are ratios with a single aggregation, rate, because a rate is already an aggregate. You compare them against a value between 0 and 1 with <, <=, >, >= or ==. Write 5% as 0.05.
The filter you choose changes what the metric covers, because rules measure only the matched spans. The Span kind filter takes values such as LLM, tool, chain and retriever. Because LLM error rate already counts only LLM calls, a Span kind filter set to LLM leaves it unchanged. Filters such as Service, Agent, Model, Step and Session narrow which LLM calls it measures. Use Span kind with Error rate when you want the share of errored spans of a different kind.
The aggregation also depends on the metric type. Numeric metrics such as Cost and Duration use per-trace semantics, so avg(Cost) is the average cost per trace in the window. Time to first token, Context usage and LLM calls are measured per LLM call, so their aggregations run over calls. Error rate and LLM error rate are the share of spans, or of LLM calls, in the window, so you do not pick an average or a percentile for them.
Set it up
- Open the custom alert builder in the Rius console and add a rule. Choose the metric LLM error rate for the share of failed LLM calls, or Error rate for the share of failed spans of the kind you filter to. The aggregation is rate.
- Set the comparison and a threshold between 0 and 1. For 5% failed calls, use
>and 0.05. - Add filters for the scope. With Error rate, add a Span kind filter and pick the kind from the suggested values. Text fields offer
=,≠,inandnot_in, soinlets you list several kinds. With LLM error rate, skip Span kind: a kind other than LLM leaves no LLM calls to measure. Add Service, Agent or Model filters to narrow the scope further. - Set the shared window, between 5 minutes and 24 hours. A longer window smooths out a few failures among few calls.
- Read the plain-language preview in the builder. It states exactly when the alert will fire, and it becomes the alert’s description.
- Attach a distribution channel (webhook, email or Slack) and choose a Renotify policy, off or every N minutes.
An alert for failed model calls from one service, and one for failed tool calls, look like this in the builder’s grammar:
FILTERS Service = checkout-agent
RULES rate(LLM error rate) > 0.05
WINDOW 15 minutes
FILTERS Span kind = tool
RULES rate(Error rate) > 0.1
WINDOW 15 minutesAn alert takes up to 10 filters with up to 50 values each and up to 10 rules. To add a second condition, add another rule. Rules combine with AND only, so for either-or behavior create a second alert. The full rule grammar is on Custom alerts.
What you get
- An alert that fires when the share of failed LLM calls in the window crosses your threshold, scoped to the service, agent, model or span kind you chose.
- A generated description of the exact firing condition on the alert’s details page.
- A firing history that records the observed value of every rule against its threshold, so you can see which condition tripped.
- Delivery to the channels you attached, with the Renotify policy you chose.
- A rule you can edit later, or delete. Deleting a custom alert removes its firing history after an explicit confirmation.
The four pre-defined alerts are described on Pre-defined alerts. Custom alerts sit beside them and cover conditions such as this one.
When something else fits
- If you want to be warned about a count of failures, not a share, use the Errors metric with count, filtered to Span kind LLM for failed model calls, in the same builder.
- If you want to catch repeated errors without choosing a threshold, the pre-defined Consistent Error Pattern alert covers that and can be turned on or off but not deleted.
- To measure whole runs rather than the matched spans, filter by Service or Agent.