How do I get alerted when my agent hits errors from the model provider?

Rius alerts you on model call errors with a custom alert on the LLM error rate metric, the share of LLM calls that errored in a rolling window. You set a threshold, scope it to a service or model, and attach a channel such as Slack, email or a webhook. When it fires, the firing page shows the observed value against your threshold and links to the affected traces. Read more in Alerts and Custom alerts.

How it works

A custom alert is one or more rules combined with AND, all evaluated over one shared rolling window. For model errors, two metrics fit:

• LLM error rate is the share of LLM calls that errored in the window. You compare it with a value between 0 and 1, so 0.05 means 5% of calls. An LLM call is an inference call, the span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count.
• Errors counts errored spans in the window, with count or ucount (distinct traces). It covers every kind of span, not only model calls.

Filters narrow which spans feed the rules before any aggregation. Use Service for one agent, Model to watch one model, or Span kind and Operation to narrow further.

Rules are evaluated every minute while the alert has no open firing, and every five minutes while one is open. A firing resolves on its own once the condition has stopped matching for two hours. If the same condition returns later, that is a new, unacknowledged firing.

Set it up

1. Open the Alerts page and add a rule in the Custom alerts section. Configuring alerts is an admin action.
2. Add a filter for the agent you care about, for example Service = checkout-agent. Add a Model filter if you only care about one model.
3. Add the error rule. Pair it with a minimum call count so a single failed call on a quiet agent does not page you:

FILTERS   Service = checkout-agent
RULES     LLM error rate > 0.05
   AND    count(LLM calls) > 20
WINDOW    15 minutes

4. Choose the window, between 5 minutes and 24 hours. A short window reacts fast. A longer one smooths over brief blips.
5. Set Renotify. Off sends one notification per firing. Every N minutes, from 5 to 1440, repeats the notification until someone acknowledges the firing or it resolves.
6. Attach a distribution channel (webhook, email or Slack) and decide whether the Root-cause analysis switch is on for this alert.
7. Save. The builder shows a plain-language sentence describing when the alert fires, and that sentence becomes the alert’s description.

There is no OR in a rule. If you want to alert on a raw error count as well as on the rate, create a second alert.

What you get

• A notification on each channel you attached as soon as the rate crosses your threshold.
• A count of open firings on the Alerts entry in the sidebar, so you see from any page that something is waiting.
• A firing page that shows when it opened, when the condition was last seen, and the observed value of every rule against its threshold, so you can tell which condition tripped.
• The affected traces, each linked to the trace view. View all in Traces opens the trace list filtered to the firing’s signature and time window, which is where you read the failing calls.
• A delivery list with one row per channel and the outcome of each attempt.
• With root-cause analysis on, a report on the firing page covering what failed, since when, and its reading of the cause.
• Acknowledgment. Ack records who saw the firing and stops further notifications for it. The firing still resolves by itself when the condition clears.

The Firings page lists every firing across the workspace. You can filter it by service, step, time window or Unacked only, and share the filtered view as a link.

When something else fits

• If you want to catch a recurring error without choosing a threshold, enable the pre-defined Consistent Error Pattern alert from the Alerts page.
• If you worry about runs that hid an error, the pre-defined Swallowed Error alert is the better fit.
• To know when ingestion stops entirely, use a custom Traces count < 1 rule over a generous window.

Related