How do I get alerted when my agent makes more LLM calls than usual?

Rius alerts you on this with a custom alert on the LLM calls metric, for example count(LLM calls) above a threshold you choose over a rolling window. Scope it to your service or agent with a filter. When it fires, the firing shows the observed count against your threshold and links to the traces behind it. You set the threshold from your normal call volume, and the rule fires when the count in the window goes above it.

How it works

A custom alert is one or more rules, combined with AND, evaluated over one shared rolling window. The LLM calls metric is an event metric, so its aggregations are count and ucount (distinct traces). The count covers inference calls only, the spans the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model.

The pieces of the rule for this question:

• Metric and aggregation: count(LLM calls) with the > operator. The other operators are <, <=, >= and ==.
• Window: between 5 minutes and 24 hours. A short window catches a sudden burst. A long window smooths out normal swings.
• Filters: Service, Agent, Model, Step, Span kind, Operation, Session or Instance. Use Service or Agent to count the calls of one agent. A Model filter counts calls to one model. Filters apply before the count, and a Step filter matches span names exactly.

Rules are evaluated every minute while the alert has no open firing, and every five minutes while one is open. The firing opens on the first match and stays open while the condition keeps matching. It resolves on its own once the condition has been clear for two hours.

The pre-defined alerts do not cover this case. Tool Loop watches repeated tool calls, and it counts only spans of kind TOOL, so it does not count LLM calls. If your agent repeats a tool, see the pre-defined detectors on the Alerts page. For a rise in model calls, build the custom alert.

Set it up

1. Open the Alerts page and create a custom alert. Creating and editing alerts is an admin action.
2. Add a filter for the agent you want to watch, for example Service or Agent, so the count covers only that agent.
3. Add the rule: metric LLM calls, aggregation count, operator >, and a value a little above your normal volume for the window. The builder shows a plain-language preview of when the alert fires. That sentence becomes the alert’s description.
4. Set the window. Set Renotify to off or to every N minutes, from 5 to 1440, if you want repeat reminders until someone acknowledges the firing.
5. Attach a distribution channel (webhook, email or Slack) so the firing reaches you.

The rule reads like this. Replace the service name, the threshold and the window with your own values:

FILTERS   Service = checkout-agent
RULES     count(LLM calls) > 200
WINDOW    15 minutes

Rules combine with AND only. If you want a second alert for a different window or a different agent, create another alert. The full builder reference is on the Custom alerts page.

What you get

• A notification on each channel attached to the alert when the call count crosses your threshold. Repeats follow your Renotify setting until you acknowledge the firing or it resolves.
• An open-firing count next to Alerts in the sidebar, so you see from any page that something is waiting. It is hidden at zero.
• A firing page that records the observed value of the rule against its threshold, when the firing opened, when the condition was last seen, and the affected traces, each linked to the trace view.
• A trace list ready to inspect. View all in Traces opens the trace list pre-filtered to the firing’s signature and time window, so you can open the runs and look at their LLM call spans.
• A delivery list with one row per channel and the outcome of each attempt.
• An optional root-cause report when root-cause analysis is on for the alert and the workspace. It says what failed, since when, and its reading of the cause.
• Acknowledgment. Acknowledging a firing records who saw it and stops further notifications. The firing still resolves on its own when the condition clears.

Saving a change to the alert resolves its open firings and starts evaluating afresh. If the condition still holds, a new firing opens under the new configuration.

When something else fits

• If the problem is the same tool being called over and over, the pre-defined Tool Loop alert fits better than a count of LLM calls.
• If you care about spend or token use rather than the number of calls, build a custom alert on Cost or the token metrics, which the Custom alerts page covers.
• If your concern is runs failing rather than running long, use an alert on Errors, Error rate or LLM error rate.

Related