Skip to Content

How do I alert on share of failed LLM calls for a single model?

In Rius, you alert on the share of failed LLM calls for one model with a custom alert that uses the LLM error rate metric and a Model filter. The filter limits the alert to calls made with that model, and the metric is the share of those LLM calls that errored in the rolling window. You compare it against a value between 0 and 1, so 0.05 means 5%.

How it works

A custom alert has optional filters, one or more rules and one shared rolling window. Filters apply first, before any aggregation, and the rules measure only the spans that match them. See Custom alerts for the full builder, and Alerts for how alerts fire.

Two parts of the alert do the work here:

  • The Model filter. It matches gen_ai.request.model on each span. It offers =, ≠, in and not_in, and it suggests values from your workspace’s live data. Use = for one model.
  • The LLM error rate metric. It is a ratio, and a rate is already an aggregate, so the builder offers only rate as its aggregation. An LLM call is an inference call, the kind of span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model.

Per-trace and per-call metrics. Numeric metrics such as Cost, Input tokens and Duration use per-trace semantics. avg(Cost) is the average cost per trace in the window, not the workspace total. Time to first token, Context usage and LLM calls are measured per LLM call, so their aggregations run over calls. Error rate and LLM error rate are shares. Error rate is the share of spans that errored, and LLM error rate is the share of LLM calls that errored. For failed model calls, use LLM error rate.

Set it up

  1. Open the custom alert builder and add a filter: Model = the model you care about, for example claude-haiku-4-5-20251001. Pick the value from the suggestions so it matches your live data exactly.
  2. Add one rule: metric LLM error rate, aggregation rate, operator >, and a value between 0 and 1. Use 0.05 to fire when more than 5% of that model’s calls fail.
  3. Set the window. It can be between 5 minutes and 24 hours, and it is shared by all rules. A longer window smooths out noise when the model gets few calls.
  4. Read the plain-language preview in the builder. It states exactly when the alert fires, and it becomes the alert’s description on the details page.
  5. Attach a channel (webhook, email or Slack) and choose the Renotify policy, off or every N minutes.

The alert reads like this:

FILTERS Model = claude-haiku-4-5-20251001 RULES rate(LLM error rate) > 0.05 WINDOW 15 minutes

To cover several models with one combined share, use in with a list of models. To watch each model separately, create one alert per model. Rules in one alert combine with AND only, so a second alert is also how you get either-or behavior.

What you get

  • An alert that fires when the share of failed LLM calls for that model passes your threshold in the window, and ignores failures from other models.
  • A firing record that shows the observed value of the rule against its threshold.
  • A details page with the generated description, the editable rule, the attached channels and the firings.
  • A custom alert you can delete later. Deleting it removes its firing history after you confirm.

When something else fits

  • If you want the share of errored spans of every kind, not only model calls, use Error rate instead of LLM error rate.
  • If you want a count of failures rather than a share, use the Errors metric with count. A fixed count suits low-traffic models better than a percentage.
  • If you want the failures of one whole agent run, filter by Service or Agent instead of Model.

Still stuck? Get help

Last updated on