Skip to Content

How do I alert on error rate for one step?

In Rius, create a custom alert with a Step filter set to the step’s span name and a rule on Error rate. The filter narrows the alert to that step’s spans before the rate is calculated, so the alert fires on the share of that step’s spans that errored in the window. You compare the rate against a value between 0 and 1, for example 0.05 for 5%.

How it works

A custom alert has optional filters, one or more rules and one rolling window. Filters apply first and select which spans feed the rules. Rules then measure only the matched spans.

  • The filter this needs is Step. A Step filter matches the span name exactly. Text filters offer =, ≠, in and not_in, and values are suggested from your workspace’s live data.
  • Error rate is a ratio. It is the share of spans that errored in the window, among the spans your filters matched. Its only aggregation is rate, because a rate is already an aggregate. LLM error rate works the same way but counts only LLM calls.
  • Other metrics aggregate differently. Numeric metrics such as Cost and Duration are calculated per trace, so avg(Cost) is the average cost per trace, not a workspace total. Time to first token, Context usage and LLM calls are calculated per LLM call instead. Error rate is neither: it counts matched spans.
  • Step names can change. The 1.0.0 SDKs name unnamed tool, agent, retrieval and model spans after their operation and target, such as execute_tool search_orders. If your agents upgrade from an earlier SDK, update the Step filter to the new name.

Set it up

  1. Open the custom alert builder. The custom alerts page describes every field.
  2. Add a filter on Step and enter the exact span name, for example execute_tool search_orders. To cover several steps in one alert, use in with a list of names. You can set up to 10 filters with up to 50 values each.
  3. Add a rule on Error rate with the rate aggregation, a comparison operator (<, <=, >, >= or ==) and a value between 0 and 1.
  4. Set the window. It can be between 5 minutes and 24 hours, and every rule shares it.
  5. Check the plain-language preview. It states exactly when the alert fires, and it becomes the alert’s description on its details page.
  6. Attach the channels you want notified.

The finished alert reads like this. The names and values are examples, so replace them with your own:

FILTERS Step = execute_tool search_orders RULES Error rate > 0.05 (aggregation: rate) WINDOW 15 minutes

To require more than a rate, add a second rule. Rules combine with AND only, so the alert fires when both hold. For example, count(Errors) > 10 next to the rate rule makes the alert ignore a single failure in a quiet window. If you want either-or behavior, create a second alert, because there is no OR.

What you get

  • An alert that watches one step’s failure share instead of the whole agent’s, so a failing tool does not get averaged away by healthy steps.
  • A threshold on a 0 to 1 scale that you can tune per step, with a window that suits how often the step runs.
  • A description generated from your rule, so anyone opening the alert sees when it fires.
  • Firings that record the observed value of every rule against its threshold, so with several rules you can see which condition tripped.
  • Delivery through the channels you attach to the alert. See Alerts for how firings, channels and renotifying fit together.

When something else fits

  • To measure whole runs rather than one step, filter by Service or Agent instead of Step.
  • If the step is a model call and you care only about failed inference calls, use LLM error rate. It counts only LLM calls.
  • If you want the failure signal without building a rule, the pre-defined alerts cover common error patterns and need no setup.

Still stuck? Get help

Last updated on