How do I alert on error rate for a single model?

In Rius, you create a custom alert with a Model filter set to that model and a rule on LLM error rate or Error rate. The filter selects only the spans that carry the model before the rate is calculated. The rule compares the share of those spans or calls that errored against a value between 0 and 1, so the alert fires only when that one model is failing.

How it works

A custom alert has optional filters, one or more rules combined with AND, and one rolling window shared by all rules. Filters apply first. They choose which spans feed the rules, and the rules then measure only those spans. The full builder is described in Custom alerts, and the wider alert system is covered in Alerts.

Two things matter for this question.

• The filter. The Model filter matches gen_ai.request.model on each span. Use = for one model. Use in to list several models in one alert, which pools them into a single rate. Values are suggested from your workspace’s live data, so you can pick the exact name your traffic uses.
• The metric. Rius offers two ratio metrics, each with a single aggregation called rate:
  • LLM error rate is the share of LLM calls that errored in the window. An LLM call is an inference call, a span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model.
  • Error rate is the share of all matched spans that errored in the window.

With a Model filter, LLM error rate answers “how often does this model’s inference fail?”. Error rate counts every span that carries the model, not only inference calls.

Per-trace and per-call aggregation. A rate is already an aggregate, so you do not pick avg or p95 for it. The numeric metrics work differently. Cost, tokens and Duration use per-trace semantics, so avg(Cost) is the average cost per trace, not the workspace total. Time to first token, Context usage and LLM calls are measured per LLM call, so their aggregations run over calls. Error rate and LLM error rate are ratios over spans or LLM calls. You compare them against a value from 0 to 1, where 0.05 means 5%.

Set it up

1. Open the custom alert builder and add a filter: Model = the model name, for example the value your code sends as the request model. To cover several models in one alert, use in and list them.
2. Add a rule on LLM error rate with the comparison > and a value between 0 and 1. Pick Error rate instead if you want every span that carries the model counted.
3. Set the window. It runs from 5 minutes to 24 hours, and every rule shares it. A short window reacts fast. A longer one smooths out a handful of failed calls.
4. Read the plain-language preview in the builder. It states exactly when the alert fires, and it becomes the alert’s description on its details page.
5. Attach a channel (webhook, email or Slack) and set Renotify to off or every N minutes.

The finished alert reads like this:

FILTERS   Model = your-model-name
RULES     rate(LLM error rate) > 0.05
WINDOW    15 minutes

This fires when more than 5% of that model’s LLM calls errored over the last 15 minutes.

To avoid alerts from a tiny sample, add a second rule with AND, such as count(LLM calls) > 20. Both rules must hold before the alert fires. Rules combine with AND only, with no OR and no nesting. If you want either-or behavior, create a second alert.

An alert takes up to 10 filters with up to 50 values each, up to 10 rules, and the comparison operators <, <=, >, >= and ==.

What you get

• An alert scoped to one model, so a failing model does not get lost in the error rate of your whole workspace.
• A firing record that stores the observed value of every rule against its threshold, so you can see which condition tripped.
• A details page with the generated description, the editable rule, the attached channels and the firings.
• A rate you can read directly as a share, because it is calculated over the matched spans or LLM calls in the window and is not averaged per trace.
• An alert you can delete when you no longer need it. Deleting it removes its firing history after an explicit confirmation.

When something else fits

• To watch failures across whole runs, filter by Service or Agent instead of Model and use Error rate. Those filters match every span that carries the value, so the rate covers the run, not one model.
• If you need different thresholds for different models, create one alert per model. A single alert with in pools its models into one rate.
• Step, Span kind and Operation filters change what a metric covers, so pick them only when you want to measure that slice.

Related