Skip to Content

How do I alert on number of LLM calls for a single agent?

Rius lets you alert on the number of LLM calls for one agent with a custom alert that filters Agent to that agent and has a rule on count of LLM calls. The alert counts only inference calls, the spans the trace view badges as LLM, over a rolling window you choose. Agent, tool, retrieval and embedding spans never count, even when they carry a model.

How it works

A custom alert has optional filters, one or more rules and one shared window. Filters apply first and decide which spans feed the rules. Then the rules aggregate those spans over the rolling window.

  • The filter this needs is Agent. It matches the agent identity Rius derives, as each span carries it. Because it matches each span’s own value, it selects every span that carries the agent, not only root spans. The LLM calls metric then counts the LLM spans among them.
  • LLM calls is an event metric. It offers two aggregations: count, the number of calls in the window, and ucount, the number of distinct traces that contain at least one call.
  • Per-trace and per-call metrics aggregate differently. Numeric metrics such as Cost, Input tokens, Output tokens, Reasoning tokens and Duration use per-trace semantics. avg(Cost) is the average cost per trace in the window, not the workspace total. Time to first token, Context usage and LLM calls are the exception. They are measured per LLM call, so their aggregations run over calls, not traces.
  • Each filter matches one field. Text fields such as Agent offer =, ≠, in and not_in, and Rius suggests values from your workspace’s live data.
  • Rules are compared with <, <=, >, >= or ==. If you add more rules to the same alert, they combine with AND only. For either-or behavior, create a second alert.

The window runs from 5 minutes to 24 hours. An alert takes up to 10 filters with up to 50 values each, and up to 10 rules.

Set it up

  1. Open the custom alert builder, the page described in Custom alerts.
  2. Add a filter: set the field to Agent and the value to your agent’s name. Pick it from the suggested values so it matches what your spans carry.
  3. Add a rule: choose the LLM calls metric, the count aggregation, the > operator and your threshold.
  4. Set the window. A shorter window catches a runaway loop quickly. A longer one smooths out normal bursts.
  5. Check the plain-language preview. It states exactly when the alert fires, and it becomes the alert’s description.
  6. Attach a channel (webhook, email or Slack) and choose a Renotify policy, off or every N minutes.

The finished alert reads like this, with your own agent name and threshold:

FILTERS Agent = checkout-agent RULES count(LLM calls) > 200 WINDOW 15 minutes

This fires when checkout-agent makes more than 200 LLM calls inside a 15 minute window. To alert on how many runs contain a call instead, use ucount with the same filter.

A custom alert is one of the alert types described under Alerts.

What you get

  • An alert scoped to one agent, so a busy neighbor in the same service does not trip it.
  • A count of real model calls only. Tool calls, retrieval, embedding and agent spans are left out, so the number matches how many times your agent called a model.
  • A choice between total calls (count) and the number of traces that made at least one call (ucount).
  • A firing history on the alert’s details page. Each firing records the observed value of every rule against its threshold, so you can see which condition tripped.
  • Notifications through the webhook, email or Slack channels you attach.
  • The freedom to edit the rule later, or delete the alert. Deleting removes its firing history after an explicit confirmation.

When something else fits

  • To watch a whole service rather than one agent, filter Service instead of Agent, or list several agents with in.
  • To alert on how many runs happened rather than how many model calls, use the Traces metric. To catch ingestion stopping entirely, a Traces count < 1 rule over a generous window fires when nothing arrives.
  • If your concern is spend rather than volume, alert on Cost with a Model filter or a Span kind of LLM. Cost is on model-call spans, and models with no known price carry no cost.

Still stuck? Get help

Last updated on