How do I get an alert on my agent’s share of failed LLM calls?

Rius alerts on your agent’s share of failed LLM calls with a custom alert on the LLM error rate metric, using the rate aggregation. You compare it against a value between 0 and 1, so 0.05 means 5% of LLM calls errored in the window. Scope the alert to your agent with a filter, pick a window from 5 minutes to 24 hours, and attach a channel to receive each firing.

How it works

LLM error rate is the share of LLM calls that errored in the rule’s window. An LLM call is an inference call, the span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model. A failing tool call does not move this metric. The Error rate metric covers all spans instead.

• Aggregation: LLM error rate is a ratio metric, so the only aggregation is rate. A rate is already an aggregate, so there is no avg or percentile to choose.
• Threshold: a value between 0 and 1, with the operators <, <=, >, >= and ==.
• Window: a rolling window from 5 minutes to 24 hours, shared by every rule in the alert.
• Filters: narrow the calls that feed the rule before the rate is computed, for example Service, Agent or Model. Each filter matches the value on the span itself, and an alert takes up to 10 filters.
• Combining rules: an alert can hold up to 10 rules, always joined with AND. There is no OR, so make a second alert for either-or conditions.

Rius evaluates a rule every minute while it has no open firing and every five minutes while one is open, so timings are approximate to that cadence.

Set it up

1. Open the Alerts page and create a custom alert. Creating and editing alerts is an admin action. Members can see firings and acknowledge them.
2. Add a filter for your agent, for example Service = your service name. Values are suggested from your workspace’s live data. Add a Model filter too if you only care about one model.
3. Add the rule LLM error rate with the rate aggregation, an operator of > and a threshold such as 0.05.
4. Add a second rule on LLM calls with count, so the alert ignores windows with very few calls. One failed call out of two is a 50% rate, and the volume rule keeps that from firing.
5. Set the window. A 15 minute window is a reasonable start for a busy agent. Use a longer window for a quiet agent, so each reading covers enough calls.
6. Attach a distribution channel (webhook, email or Slack) and choose a Renotify policy: off, or every N minutes from 5 to 1440 until someone acknowledges the firing or it resolves.

The values from steps 2 to 5 look like this. The service name, threshold, call count and window are examples to replace with your own:

Filter   Service = checkout-agent
Rule 1   LLM error rate, rate > 0.05
Rule 2   LLM calls, count >= 20
Window   15 minutes

Both rules must hold at once for the alert to fire. The builder shows a plain-language preview of when the alert fires, and that sentence becomes the alert’s description. Before you save, know that saving a change later resolves the alert’s open firings and starts evaluation afresh. For the full builder, see Custom alerts.

What you get

When the rule matches, Rius opens a firing and delivers it to every channel attached to the alert. The firing records the observed value of each rule against its threshold, so you can see whether the error rate or the volume rule tripped. Its own page shows when it opened, when the condition was last seen, the affected traces linked to the trace view, and one delivery row per channel with the outcome of each attempt.

• Acknowledgment: acknowledging a firing records who saw it and stops further notifications for it.
• Resolution: once the condition stops matching for two hours, the firing resolves on its own. If the condition returns, that is a new, unacknowledged firing.
• Open count: the Alerts entry in the sidebar counts open firings nobody has acknowledged.
• Root-cause analysis: switch it on for the alert and, when the workspace has it enabled, each firing is handed to the analysis agent, which writes a report onto the firing page.

The Alerts page covers firings, renotify and rule health in full.

When something else fits

• To watch failures across every span, not only model calls, use the Error rate metric. To watch an absolute number of failures, use the Errors metric with count or ucount.
• For a recurring error, a run that swallowed an error, or an agent stuck in a loop, the pre-defined alerts Consistent Error Pattern, Swallowed Error and Tool Loop need no rule of your own.

Related