Skip to Content

How do I alert on the share of failed LLM calls for one step?

Rius lets you alert on the share of failed LLM calls for one step with a custom alert that uses the LLM error rate metric and a Step filter. The filter narrows the alert to spans with that step’s name, and the metric is the share of those LLM calls that errored in the window. You compare it against a value between 0 and 1, and the alert fires when the share crosses it.

How it works

A custom alert is one or more rules, combined with AND, evaluated over one shared rolling window. Filters apply first. They select which spans feed the rules, before any aggregation. Each filter matches a span’s own value, so the rules measure only the spans that match.

Two parts of this question matter.

  • The filter. Step matches the span name exactly. To scope the alert to one step, set Step to that name. Values are suggested from your workspace’s live data, so you can pick the name instead of typing it. The Step filter counts LLM calls only when the step’s span is itself an LLM call, the kind the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count as LLM calls. If you name a wrapping chain or agent span, the alert has no LLM calls to measure, so choose the name of the model-call span.
  • The metric. LLM error rate is a ratio, and its only aggregation is rate, because a rate is already an aggregate. It is the share of LLM calls that errored in the window, so it is computed over calls across the whole window, not averaged per trace.

That last point differs from the numeric metrics. Cost, tokens and Duration use per-trace semantics: avg(Cost) is the average cost per trace in the window. Time to first token, Context usage and LLM calls are the exception, because they are measured per LLM call. LLM error rate follows the per-call view, so a trace with many failed calls weighs more than a trace with one.

Comparison operators are <, <=, >, >= and ==. The window runs from 5 minutes to 24 hours. An alert takes up to 10 filters with up to 50 values each, and up to 10 rules.

Set it up

  1. Open the custom alert builder and add a filter: Step, operator =, and the name of the span for the model call you care about. Use in if the step has more than one span name.
  2. Add a rule on the LLM error rate metric with the operator > and a threshold between 0 and 1. For example, 0.05 fires when more than 5% of the step’s LLM calls fail.
  3. Add a second rule on LLM calls with count, so the alert ignores windows with only a few calls. Rules combine with AND, and this rule measures only the matched spans too, so the count is for that step.
  4. Set the window. A longer window smooths out noise on a step that runs rarely.
  5. Check the plain-language preview in the builder. It shows exactly when the alert fires and becomes the alert’s description.
  6. Attach a channel and save. Renotify can be off or every N minutes.

The finished alert reads like this:

FILTERS Step = summarize-ticket RULES LLM error rate > 0.05 AND count(LLM calls) > 20 WINDOW 15 minutes

summarize-ticket is an example name. Use your own span name. If you upgraded to the 1.0.0 SDKs, unnamed model spans are named after their operation and target, so an alert written against an older name needs the new one. For the full list of fields, metrics and limits, see Custom alerts. For how alerts fire and are routed, see Alerts.

What you get

  • An alert on the failure share of one step’s LLM calls, not on errors across the whole workspace.
  • A threshold that stays meaningful as volume changes, because it is a share between 0 and 1 and not a raw count.
  • Protection against low-volume noise when you pair the rate with a minimum call count.
  • A details page with the generated description, the editable rule, attached channels and firings. Firings record the observed value of every rule against its threshold, so you can see which condition tripped.

When something else fits

  • To alert on failures across a whole run, use the Error rate metric with a Service or Agent filter instead of a Step filter. It is the share of spans that errored.
  • To alert on all LLM failures for one model, filter Model, or filter Span kind to LLM, rather than naming a step.
  • For either-or conditions, create a second alert, because rules combine with AND only.

Still stuck? Get help

Last updated on