Skip to Content

How do I get an alert on my agent’s time to first token?

In Rius, you create a custom alert with a rule on the Time to first token metric, such as p95 above a threshold in milliseconds. Rius evaluates the rule over a rolling window you choose. When it holds, Rius opens a firing and delivers it to every channel attached to the alert. The Custom alerts builder shows a plain-language preview of when the alert fires.

How it works

Time to first token is one of the metrics a custom alert can watch. It is a numeric metric in milliseconds, and the builder offers avg, min, max, p50, p95 and p99 as aggregations.

  • It is measured per LLM call. The aggregation runs over inference calls, the spans the trace view badges as LLM, not over whole traces. Agent, tool, retrieval and embedding spans never count, even if they carry a model.
  • It needs streamed calls. The metric exists on streamed LLM calls whose instrumentation records it. Calls without a recorded value are not part of the reading.
  • Rules share one window. A custom alert holds one or more rules, combined with AND only, over one rolling window of 5 minutes to 24 hours.
  • Filters narrow the spans first. You can scope by Service, Agent, Step, Model, Span kind, Operation, Session or Instance before the aggregation runs.
  • Evaluation cadence. Rius evaluates a rule every minute while it has no open firing, and every five minutes while one is open, so timing is approximate to that cadence.

Set it up

  1. Open the Alerts page and start a custom alert. Creating and editing alerts is an admin action.
  2. Optionally add a filter so the alert watches the agent you care about, for example Service = checkout-agent. Use a Model filter if one model is the one you worry about.
  3. Add a rule on Time to first token and choose p95. Slow starts show up in the tail first, so p95 catches them while avg can hide them. Use p99 for a stricter watch, or avg if you want to track the typical call.
  4. Set the threshold in milliseconds, above what your agent normally reaches. Check the p50 and p95 you see today on the Usage and cost view, then set the threshold clearly above the p95 you consider healthy.
  5. Set the window. 15 minutes is a reasonable starting point: long enough that a few slow calls do not trip it, short enough to react quickly. A very short window over few calls makes a percentile jumpy.
  6. Optionally add a second rule with AND so the alert only fires when there is enough traffic to trust the percentile. The example below uses LLM calls for that.
  7. Attach one or more distribution channels (webhook, email or Slack), and choose a Renotify policy: off for one notification per firing, or every 5 to 1440 minutes until someone acknowledges it.
FILTERS   Service = checkout-agent
RULES     p95(Time to first token) > 2000
   AND    count(LLM calls) > 20
WINDOW    15 minutes

The threshold of 2000 is 2 seconds, because the unit is milliseconds. Replace the service name and the numbers with your own.

What you get

  • A firing when the condition holds. Each firing records the observed value of every rule against its threshold, so you can see which condition tripped.
  • Delivery to every attached channel. The firing page lists one row per channel with the outcome of each attempt, and Renotify repeats the notification until a member acknowledges the firing or it resolves.
  • A coverage ratio. Firings on time to first token show how much of the window’s calls carried a reading, so you can tell a real reading from a partly covered one.
  • A path to the cause. The firing page links to the affected traces and opens the trace list for the firing’s time window.
  • A clean lifecycle. The firing resolves on its own after the condition stops matching for two hours. Acknowledging it stops further notifications. Any member can acknowledge, and saving a change to the alert resolves its open firings and starts evaluation afresh.

Coverage is attached only to firings. A rule held under its threshold by low coverage stays quiet. For the full alerting model, see Alerts.

When something else fits

  • If you want a view of time to first token over time, not a trigger, the p50 and p95 summary on the Usage and cost view fits better.
  • If you need either-or logic, such as slow starts or high errors, create a second alert, because rules in one alert combine with AND only.
  • If your calls are not streamed, the metric has no readings, so alert on Duration instead.

Still stuck? Get help

Last updated on
Quickstart