How do I alert on error rate for a single agent?

Rius lets you alert on one agent’s error rate with a custom alert that filters Agent to that agent and sets an Error rate rule against a value between 0 and 1. The filter narrows the alert to spans that carry that agent, and the rule fires when the share of those spans that errored in the window crosses your threshold. Error rate is a ratio over spans, so it does not follow the per-trace averaging that numeric metrics such as cost use.

How it works

A custom alert is one or more rules, combined with AND, evaluated over one shared rolling window. Filters apply first and decide which spans feed the rules. Aggregation comes after.

• The Agent filter. It matches the agent identity Rius derives, as each span carries it. Every span that carries the value is selected, not only root spans, so the alert covers the agent’s model calls, tool calls and steps.
• Error rate. It is a ratio metric. Its only aggregation is rate, because a rate is already an aggregate. It is the share of matched spans that errored in the window, shown from 0 to 1, so 0.05 means 5%.
• LLM error rate. It works the same way but counts only LLM calls, the inference calls the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model.
• Per-trace and per-call metrics. Numeric metrics such as Cost, Duration and the token metrics use per-trace semantics. avg(Cost) is the average cost per trace in the window, not the workspace total. Time to first token, Context usage and LLM calls are measured per LLM call, so their aggregations run over calls. Error rate is neither of these: it is a share of spans, or of LLM calls for LLM error rate.
• Other filters change coverage. Step, Model, Span kind and Operation filters change what a metric covers. For an agent-wide error rate, use the Agent filter alone, or with Service. Use Service or Agent to measure whole runs.

Comparison operators are <, <=, >, >= and ==. The window runs from 5 minutes to 24 hours.

Set it up

1. Open the custom alert builder, as described on Custom alerts.
2. Add a filter with the field Agent and the operator =. Pick your agent from the suggested values, which come from your workspace’s live data. Use in to cover several agents with one alert, or not_in to exclude some.
3. Add a rule on Error rate with the aggregation rate, the operator > and a value between 0 and 1. For example, 0.05 fires when more than 5% of the agent’s spans errored in the window.
4. Optionally add a second rule. Rules combine with AND only, so the alert fires only when both hold. A count rule on Errors keeps a handful of spans on a quiet agent from tripping the rate.
5. Set the shared window, from 5 minutes to 24 hours. A short window reacts fast and a long one smooths out spikes.
6. Attach a channel (webhook, email or Slack) and choose the Renotify policy, off or every N minutes. Check the plain-language preview, which becomes the alert’s description.

The rule then reads like this:

FILTERS   Agent = checkout-agent
RULES     rate(Error rate) > 0.05
   AND    count(Errors) > 10
WINDOW    15 minutes

Replace checkout-agent with your own agent name. The builder shows the exact wording for the rule it generates. For model-call failures only, swap the first rule for LLM error rate.

What you get

• An alert scoped to one agent, so another agent’s failures do not move the number you watch.
• A threshold on the share of spans that errored, not on a raw count, so it holds up as traffic rises and falls.
• A details page with the generated description, the editable rule, attached channels and firings. Each firing records the observed value of every rule against its threshold, so you can see which condition tripped.
• A custom alert you can delete, which removes its firing history after an explicit confirmation.
• One alert per either-or case. If you want to alert when either of two conditions holds, create a second alert, because there is no OR.

For how alerts fire, renotify and reach channels, see Alerts.

When something else fits

• If you want errors from one kind of step, such as tool calls, add a Span kind or Step filter. The rate is then over those spans only, not the whole run.
• If you only care about failed model calls, use LLM error rate instead of Error rate.
• If you want a plain count of failures rather than a share, use the Errors metric with count or ucount, where ucount counts distinct traces.

Related