How do I alert on time to first token for a single agent?

Rius lets you alert on time to first token for one agent with a custom alert that has an Agent filter and a rule on the Time to first token metric. For example, p95(Time to first token) > 1500 fires when the slowest 5% of that agent’s model calls in the window take longer than 1,500 ms to return their first token. Time to first token is measured per model call, so the aggregation runs over calls, not traces.

How it works

A custom alert is one or more rules combined with AND, all evaluated over one shared rolling window. Optional filters narrow which spans feed the rules, and they apply before any aggregation.

The filter this needs. Use the Agent filter. It matches the agent identity Rius derives, as each span carries it, so it selects every span that carries that agent, not only root spans. The Agent filter offers =, ≠, in and not_in, and it suggests values from your workspace’s live data. Use in to cover several agents with one alert.

How the metric aggregates. Most numeric metrics, such as Cost, Duration and the token counts, use per-trace semantics: avg(Cost) is the average cost per trace in the window, not a workspace total. Time to first token is an exception, along with Context usage and LLM calls. These are measured per model call. An LLM call is an inference call, a span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model.

Time to first token offers these aggregations:

• avg
• min
• max
• p50
• p95
• p99

The unit is milliseconds. Percentiles are a good fit for a latency metric. p95 tells you how slow the worst calls are, while avg can hide them.

Why Agent and not Step or Span kind. Step, Model, Span kind and Operation filters change what a metric covers, because the rules measure only the matched spans. Time to first token lives on model calls, so a filter that selects only tool, chain or agent spans leaves nothing to measure. The Agent filter keeps every span that agent produced, including its model calls, so the metric has data to read.

Coverage. Time to first token exists only on streamed model calls whose instrumentation records it. Calls without it are left out of the rule. Firings show the coverage ratio, so you can tell a real reading from a partly covered one. A rule held under its threshold by low coverage stays quiet, so confirm your agent streams its responses before you rely on the alert.

Set it up

1. Open the custom alert builder from Alerts and start a new custom alert.
2. Add a filter on Agent with = and the name of your agent. Pick it from the suggested values. To cover several agents, choose in and list them.
3. Add a rule on Time to first token. Choose an aggregation, such as p95, a comparison operator and a threshold in milliseconds.
4. Set the window. It is shared by all rules and runs from 5 minutes to 24 hours.
5. Attach a channel (webhook, email or Slack) and choose the Renotify policy: off, or every N minutes.
6. Read the plain-language preview in the builder. It states exactly when the alert fires, and it becomes the alert’s description on its details page.

The finished alert reads like this, with your own agent name and threshold:

FILTERS   Agent = checkout-agent
RULES     p95(Time to first token) > 1500
WINDOW    15 minutes

To add a second condition, add another rule. Rules combine with AND only, with no OR and no nesting. If you want either-or behavior, create a second alert. An alert takes up to 10 filters with up to 50 values each, and up to 10 rules. The full grammar and metric table are on the Custom alerts page.

What you get

• An alert scoped to one agent, so a slow agent does not hide behind the numbers of the rest of your workspace.
• A latency reading taken per model call, so a p95 describes the calls your users wait on and is not diluted by tool or chain spans.
• A firing record that shows the observed value of every rule against its threshold, plus the coverage ratio for time to first token.
• Notifications through webhook, email or Slack, with optional renotification while the alert keeps firing.
• A details page with the generated description, the editable rule, the attached channels and the firings. You can delete a custom alert after an explicit confirmation, which also removes its firing history.

When something else fits

• If you want one alert across every agent in a service, filter on Service instead of Agent. Use Service or Agent whenever you want to measure whole runs.
• If your agent does not stream responses, or its instrumentation does not record time to first token, alert on Duration instead, which is measured per trace.
• If you want to look at time to first token without a trigger, the usage and cost summary shows it as p50 and p95 over a time range.

Related