How do I alert on time to first token for one span kind?

Rius lets you alert on time to first token for one span kind by adding a Span kind filter, set to LLM, to a custom alert with a rule on Time to first token. The filter selects which spans feed the rule before any aggregation. Time to first token is measured per LLM call, so the rule’s aggregation, such as p95, runs over calls rather than traces.

How it works

A custom alert in Alerts is one or more rules, combined with AND, evaluated over one shared rolling window. Optional filters narrow which spans the rules see. They apply before aggregation.

• The filter you need. Span kind is one of the filter fields. It matches the kind of each span, such as LLM, tool, chain or retriever. Text fields offer =, ≠, in and not_in, and values are suggested from your workspace’s live data.
• How the metric aggregates. Time to first token offers avg, min, max, p50, p95 and p99, in milliseconds. It is one of three metrics measured per LLM call, along with Context usage and LLM calls. The other numeric metrics (Cost, the token metrics and Duration) aggregate per trace, so avg(Cost) is the average cost per trace, not the workspace total.
• What counts as a call. An LLM call is an inference call, a span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model. A Span kind filter on LLM therefore matches what the metric already measures, and it keeps the alert explicit and easy to read.
• A filter on another kind measures nothing. If you filter Span kind to tool, chain or agent, no LLM calls match, so the rule has no time to first token to evaluate.
• Where the value exists. Time to first token exists only on streamed LLM calls whose instrumentation records it. Calls without it are not part of the reading.
• Duration behaves differently. If you add a rule on Duration to the same alert, a Span kind filter changes what it covers: per trace, it runs from the earliest matched span’s start to the latest matched span’s end, not across the whole run.

Set it up

1. Open the custom alert builder and add a filter on Span kind with the value LLM. To watch one model instead, or as well, add a Model filter, which matches gen_ai.request.model. To limit the alert to one agent, add a Service or Agent filter.
2. Add a rule on Time to first token. Pick an aggregation: p95 or p99 catches slow tails, while avg smooths them out. Choose a comparison operator and a threshold in milliseconds.
3. Set the window. It is shared by all rules and runs from 5 minutes to 24 hours.
4. Check the plain-language preview. It states exactly when the alert fires, and it becomes the alert’s description on its details page.
5. Attach a channel (webhook, email or Slack) and choose a Renotify policy: off, or every N minutes.

The finished alert reads like this:

FILTERS   Span kind = LLM
RULES     p95(Time to first token) > 2000
WINDOW    15 minutes

This fires when the slowest 5% of LLM calls in the window take more than 2000 ms to produce their first token. Rules combine with AND only. If you want either-or behaviour, for example one threshold for streamed chat calls and another for a different model, create a second alert.

An alert takes up to 10 filters with up to 50 values each, and up to 10 rules. See Custom alerts for the full filter and metric reference.

What you get

• An alert that watches the first-token latency of your model calls only, with no tool, chain or retrieval spans involved.
• Aggregations over calls, so a p95 reflects the slowest 5% of individual calls in the window, not the slowest traces.
• A firing record with the observed value of every rule against its threshold, so you see which condition tripped.
• Coverage on each firing. Firings on Time to first token show the coverage ratio, so you can tell a real reading from one based on few streamed calls. Coverage is attached only to firings. A rule held under its threshold by low coverage stays quiet.
• A custom alert you can edit and delete. Deleting removes its firing history after an explicit confirmation.

When something else fits

• If you want to watch whole runs rather than model calls, filter by Service or Agent instead of Span kind, and use a per-trace metric such as Duration.
• If your calls are not streamed, or your instrumentation does not record time to first token, a rule on Time to first token has nothing to measure. Duration on an LLM-filtered alert is the closer choice.

Related