How do I alert on run duration for one span kind?

Rius lets you do this with a custom alert that has a Span kind filter and a rule on the Duration metric, such as p95(Duration) > 5000. With a Span kind filter, Duration is measured per trace from the earliest matched span’s start to the latest matched span’s end. It is not the length of the whole run. To alert on whole-run duration, filter by Service or Agent instead.

How it works

A custom alert is one or more rules, combined with AND, evaluated over one shared rolling window. Filters apply first and select which spans feed the rules. Aggregation comes after.

• Filters decide what Duration covers. Rules measure only the matched spans. With a Span kind filter (or a Step, Model or Operation filter), Duration runs, per trace, from the earliest matched span’s start to the latest matched span’s end. Any gap between matched spans in that stretch counts toward the value.
• Duration is a per-trace metric. It offers avg, min, max, p50, p95 and p99, in milliseconds. p95(Duration) is the 95th percentile across traces in the window, not across individual spans.
• Some metrics are per call instead. Time to first token, Context usage and LLM calls are measured per LLM call, so their aggregations are over calls, not traces. An LLM call is a span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model.
• Span kind values include LLM, tool, chain and retriever. Text filters offer =, ≠, in and not_in, and values are suggested from your workspace’s live data.
• Service and Agent filters match every span that carries the value, not only root spans. Use them when you want the duration of whole runs.

Set it up

1. Open the custom alert builder. The custom alerts page describes it in full, and Alerts covers how alerts fire and renotify.
2. Add a Span kind filter, for example Span kind = tool. To cover several kinds, use in with the list. If you also want to limit the alert to one agent, add a Service or Agent filter. Filters combine, and an alert takes up to 10 filters with up to 50 values each.
3. Add a rule on Duration. Pick an aggregation (avg, min, max, p50, p95 or p99), a comparison operator (<, <=, >, >= or ==) and a threshold in milliseconds.
4. Set the window, between 5 minutes and 24 hours. All rules in the alert share it.
5. Read the plain-language preview. It says exactly when the alert fires, and it becomes the alert’s description on its details page.
6. Attach a channel and save.

The alert from these steps looks like this in the rule grammar:

FILTERS   Span kind = tool
RULES     p95(Duration) > 5000
WINDOW    15 minutes

This fires when, across traces in the last 15 minutes, the 95th percentile of the tool-span stretch exceeds 5,000 ms. Add more rules, such as a count of Errors, and they must all hold at once, because rules combine with AND only. For either-or behaviour, create a second alert.

What you get

• An alert on how long one kind of step takes, measured from the first matched span to the last matched span in each trace.
• A choice of aggregation, so you can alert on typical duration with avg or p50, or on the slow tail with p95 or p99.
• Firings that record the observed value of every rule against its threshold, so you can see which condition tripped.
• A generated description on the alert’s details page, taken from the builder’s preview.
• A custom alert you can edit and delete. Deleting it removes its firing history after an explicit confirmation.

When something else fits

• To alert on how long whole runs take, filter by Service or Agent rather than Span kind, so Duration spans the full run.
• To alert on how long model calls take to start answering, use Time to first token, which is measured per LLM call. It exists only on streamed calls whose instrumentation records it.
• To get notified if ingestion stops entirely, use a Traces count < 1 rule over a generous window.

Related