Skip to Content

How do I alert on error rate for one service?

Rius lets you alert on error rate for one service with a custom alert that has a Service filter and an Error rate rule. The filter picks the spans of that service before the rate is calculated, so other services never affect it. Error rate is a share between 0 and 1, so you compare it against a value such as 0.05 for 5%.

How it works

A custom alert is a threshold on a workspace metric, optionally narrowed by filters. The Custom alerts page covers the full builder, and the Alerts page covers how alerts fire and notify.

Two parts matter for this question.

The filter. Filters choose which spans feed the rules, before any aggregation. The Service filter matches service.name, the name your instrumentation sends. Filters on text fields offer =, ≠, in and not_in, and the builder suggests values from your workspace’s live data. Use = for one service. Use in to cover several services with one alert.

The metric. Error rate is a ratio metric, and its only aggregation is rate, because a rate is already an aggregate. It is the share of spans that errored in the window. LLM error rate is the same idea, but counts only LLM calls, meaning inference calls that the trace view badges as LLM. Agent, tool, retrieval and embedding spans are not counted there, even when they carry a model.

This differs from other metrics. Numeric metrics such as Cost, Duration and token counts aggregate per trace. avg(Cost) is the average cost per trace in the window, not the workspace total. Time to first token, Context usage and LLM calls are measured per LLM call, so their aggregations run over calls. Error rate sits apart from both: it counts spans, or LLM calls for LLM error rate, that errored out of the spans that matched your filter.

Because the rule measures only matched spans, the filter you choose changes what the rate covers. A Service filter alone includes every span of that service, so the rate covers the whole service. Adding a Step, Model, Span kind or Operation filter narrows the rate to those spans.

Set it up

  1. Open the custom alert builder and add a filter with the field Service, the operator = and the name of your service. The builder suggests names it has seen in your data.
  2. Add a rule on Error rate. Choose a comparison operator (<, <=, >, >= or ==) and a value between 0 and 1. For example, > with 0.05 fires when more than 5% of the service’s spans errored.
  3. Set the window. It is shared by all rules and can be between 5 minutes and 24 hours. A short window reacts faster. A longer one smooths out brief spikes.
  4. Check the plain-language preview. It states exactly when the alert fires, and it becomes the alert’s description on its details page.
  5. Attach a channel (webhook, email or Slack) and choose a Renotify policy, either off or every N minutes.

The alert reads like this:

FILTERS Service = checkout-agent RULES Error rate > 0.05 WINDOW 15 minutes

To require a second condition, add another rule. Rules always combine with AND and share one window. For example, add count(Errors) > 10 so the alert fires only when the rate is high and at least 10 errors occurred. Rius has no OR. For either-or behavior, create a second alert.

An alert accepts up to 10 filters with up to 50 values each, and up to 10 rules.

What you get

  • An alert scoped to one service. Errors in other services do not move its rate.
  • A rate that you read the same way every time: errored spans divided by matched spans in the window, as a value from 0 to 1.
  • A details page with the generated description, the editable rule, the attached channels and the firings. Each firing records the observed value of every rule against its threshold, so you can see which condition tripped.
  • A custom alert you can edit or delete. Deleting it removes its firing history after you confirm.

When something else fits

  • If you care only about model failures, use LLM error rate instead of Error rate. It counts only LLM calls, so failing tool or chain spans do not affect it.
  • If you want a count of failures rather than a share, use the Errors metric with count, or ucount for distinct traces. This suits low-traffic services, where a rate swings widely on a few spans.

Still stuck? Get help

Last updated on