How do I alert on output tokens for one service?

Rius lets you alert on output tokens for one service with a custom alert that has a Service filter and a rule on the Output tokens metric. The filter limits the alert to spans from that service. The rule compares an aggregation of output tokens per trace against your threshold over a rolling window of 5 minutes to 24 hours. When the rule trips, the alert fires and sends to the channels you attach.

How it works

A custom alert is built from filters, rules and a window. Filters select which spans feed the rules, and they are applied before any aggregation.

For this alert you need one filter and one rule:

• The filter is Service. It matches service.name, the name your instrumentation sends. Suggested values come from your workspace’s live data, so you pick the service from a list. Service is also the filter to use when you want to measure whole runs.
• The rule is Output tokens. It is a numeric metric, so you choose one aggregation: avg, min, max, p50, p95 or p99. You then choose a comparison operator (<, <=, >, >= or ==) and a threshold in tokens.
• The window is shared by all rules. It runs from 5 minutes to 24 hours.

Numeric metrics aggregate per trace. avg(Output tokens) is the average output tokens per trace in the window, not the total for the service. So a threshold of 2,000 on the average means a typical trace produced more than 2,000 output tokens, and p95 means the heaviest 5% of traces did.

Three metrics behave differently: Time to first token, Context usage and LLM calls are measured per LLM call, and their aggregations run over calls. Output tokens is not one of them, so read its aggregations as per trace.

Rules measure only the spans the filters match. Output tokens are carried by model-call spans. If you add a Step, Span kind or Operation filter that selects only tool, chain or agent spans, the rule has no tokens to measure. With only the Service filter, you measure whole runs of that service.

Set it up

1. Open the custom alert builder on the alerts page. The custom alerts page describes every field.
2. Add a filter: choose Service, choose =, and pick your service name from the suggestions. To cover several services in one alert, use in with a list of values. To cover everything except one, use ≠ or not_in.
3. Add a rule: choose the Output tokens metric, an aggregation, an operator and a threshold.
4. Set the window. A short window reacts quickly. A long one smooths out single large traces.
5. Read the plain-language preview. It states exactly when the alert fires, and it becomes the alert’s description.
6. Attach a distribution channel (webhook, email or Slack) and choose a Renotify policy: off, or every N minutes.

The alert you build reads like this:

FILTERS   Service = checkout-agent
RULES     p95(Output tokens) > 4000
WINDOW    15 minutes

Replace checkout-agent with your own service name and the threshold with a value that fits your traffic.

An alert takes up to 10 filters with up to 50 values each, and up to 10 rules. Rules always combine with AND. To alert on either of two conditions, create a second alert. If you add a second rule, such as an error count, both must hold in the same window for the alert to fire.

What you get

• An alert scoped to one service, so output token spikes in other services do not trigger it.
• A threshold on the statistic you care about: the average for gradual drift, or p95 and p99 for rare heavy traces.
• A details page with the generated description, the editable rule, the attached channels and every firing. Each firing records the observed value of the rule against its threshold.
• Notifications through webhook, email or Slack, repeated at the interval you choose.
• A custom alert you can edit, and delete after an explicit confirmation. Deleting it also removes its firing history.

For how firing, acknowledgement and renotification work across all alert types, see Alerts.

When something else fits

• To alert on one agent instead of a service, filter on Agent. It matches every span that carries that agent identity.
• To watch output tokens for one model, filter Model. For one step, filter Step, which matches the span name exactly. The rule then measures only those matched spans, and Duration under these filters runs from the earliest matched span’s start to the latest matched span’s end.
• To watch the volume of model calls rather than the size of each trace, use the LLM calls metric, which counts calls.

Related