How do I alert on time to first token for a single model?

Rius lets you alert on time to first token for one model with a custom alert that has a Model filter and a rule on the Time to first token metric, such as p95(Time to first token) > 800. The Model filter matches gen_ai.request.model, so only calls to that model feed the rule. Time to first token is measured per LLM call, so the aggregation runs over calls, not traces.

How it works

A custom alert is one or more rules, combined with AND, evaluated over one shared rolling window. Filters apply first and choose which spans feed the rules. Aggregation comes after.

• The filter this needs. Use the Model filter, which matches gen_ai.request.model. It offers =, ≠, in and not_in, and suggests values from your workspace’s live data. Use = for one model. Use in for a few models that should share a threshold.
• The metric. Time to first token is a numeric metric in milliseconds. It offers avg, min, max, p50, p95 and p99.
• How it aggregates. Most numeric metrics are per trace: avg(Cost) is the average cost per trace in the window, not the workspace total. Time to first token, Context usage and LLM calls are the exception. They are measured per LLM call, so p95(Time to first token) is the 95th percentile across calls in the window. A single trace with many calls contributes each call.
• What counts as a call. An LLM call is an inference call, a span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model.
• Coverage. Time to first token exists only on streamed LLM calls whose instrumentation records it. Calls without it are left out. Firings show a coverage ratio, so you can tell a real reading from a partly covered one. A rule held under its threshold by low coverage stays quiet.

Comparison operators are <, <=, >, >= and ==. The window runs from 5 minutes to 24 hours.

Set it up

1. Open the custom alert builder from Alerts and create a new alert. The full builder reference is on Custom alerts.
2. Add a filter: field Model, operator =, and pick the model from the suggested values, for example claude-haiku-4-5-20251001.
3. Add a rule on Time to first token. Choose p95 for a tail-latency alert, or avg if you care about the typical call. Set the comparison and a threshold in milliseconds.
4. Set the window. A short window such as 15 minutes reacts quickly. A longer one smooths out a few slow calls.
5. Check the plain-language preview in the builder. It states exactly when the alert fires, and it becomes the alert’s description on the details page.
6. Attach a channel (webhook, email or Slack) and choose a Renotify policy, off or every N minutes.

The finished alert has this shape:

FILTERS   Model = claude-haiku-4-5-20251001
RULES     p95(Time to first token) > 800
WINDOW    15 minutes

The 800 and the 15 minutes are example values, so choose your own. To watch a second model on a different threshold, create a second alert. Rules inside one alert combine with AND only, and there is no OR.

To alert on several models together, change the filter to in and list them. They then share one threshold and one aggregation, so the p95 is taken across all their calls combined.

What you get

• An alert that watches only the calls to the model you chose, so a slow model does not hide behind a fast one in a workspace-wide number.
• A percentile over individual calls, so a slow tail shows up even when most calls are quick.
• Firings that record the observed value against your threshold, plus the coverage ratio, so you can see how many calls the reading rests on.
• A details page with the generated description, the editable rule, the attached channels and the firing history. You can delete a custom alert, which removes its firing history after you confirm.
• Up to 10 filters with up to 50 values each and up to 10 rules per alert, which leaves room to narrow further, for example by Service or Agent as well as Model.

A Model filter also changes what other metrics cover. With Model set, Duration runs per trace from the earliest matched span’s start to the latest matched span’s end, not across the whole run. If you want whole-run duration, filter by Service or Agent instead of Model.

When something else fits

• If your calls are not streamed, or your instrumentation does not record time to first token, there is nothing for this rule to measure. Alert on Duration with the Model filter instead, and read it as the span of the matched calls, not the whole run.
• To watch whole-run latency, filter by Service or Agent rather than Model, and use Duration.
• For pre-built checks on errors, tool loops and context, the pre-defined alerts need no setup.

Related