How do I get an alert on my agent’s number of LLM calls?

Rius lets you build a custom alert on the LLM calls metric that fires when the number of LLM calls in a rolling window crosses a threshold you set. Use the count aggregation, filter to your agent’s service, and pick a window from 5 minutes to 24 hours. Each firing is delivered to the channels you attach and records the observed value against your threshold. You build it in the custom alerts section of Alerts.

How it works

A custom alert is one or more rules combined with AND, all evaluated over one shared rolling window. For this metric, the rule looks like this:

• Metric: LLM calls. It is an event metric, so you choose between two aggregations.
• count: the number of LLM calls in the window. Use this for most alerts.
• ucount: the number of distinct traces in the window that contain LLM calls. Use it when you care how many runs made calls, not how many calls were made.
• What counts as a call: an inference call, meaning a span the trace view badges as LLM. Agent, tool, retrieval and embedding spans never count, even when they carry a model.
• Comparison: less than, less than or equal, greater than, greater than or equal, or equal to.
• Window: between 5 minutes and 24 hours.

Filters narrow which spans feed the rule before it counts. You can filter on service, agent, step, model, span kind, operation, session and instance. An alert takes up to 10 filters and up to 10 rules.

Rius evaluates a rule every minute while it has no open firing, and every five minutes while one is open, so timing is approximate to that cadence. Configuring alerts is an admin action. Members can see firings and acknowledge them. The full rule syntax is on the Custom alerts page.

Set it up

1. Open the Alerts page and start a new custom alert in the custom alerts section. You need admin rights.
2. Add a Service filter for your agent, for example checkout-agent. Add a Model filter as well if only one model should count.
3. Add a rule on LLM calls with the count aggregation, the greater than operator and a threshold taken from your normal traffic.
4. Set the window. A short window such as 15 minutes catches a burst of calls, such as a run that keeps retrying. A longer window such as 1 hour or more suits a steady budget on call volume. Keep the threshold in proportion to the window, because the count covers the whole window.
5. Attach one or more channels (webhook, email or Slack), and set Renotify to off or to a repeat interval from 5 to 1440 minutes.
6. Read the plain-language preview in the builder. It becomes the alert’s description, so check that it says what you mean, then save.

The rule from these steps, in the builder’s notation:

FILTERS   Service = checkout-agent
RULES     count(LLM calls) > 500
WINDOW    15 minutes

The 500 is a placeholder. Set it above the highest count you see on a normal day for that window. Saving any later change resolves the alert’s open firings and starts evaluation afresh, so a condition that still holds opens a new firing.

What you get

• A firing when the count crosses your threshold. It opens the first time the condition matches and stays open while it keeps matching.
• The observed value next to the threshold. Each firing records the value of every rule against its threshold, so you can see which condition tripped.
• Delivery to every attached channel. The firing’s page lists one row per channel with the outcome of each attempt, and each repeat from Renotify adds a row.
• Links to the affected traces. The firing page links the traces and opens the trace list filtered to the firing’s time window.
• Acknowledgment. Acknowledging a firing records who saw it and stops further notifications. The firing still resolves on its own after the condition has stopped matching for two hours.
• Optional root-cause analysis. Turn on the alert’s Root-cause analysis switch and the firing gets a report on what failed and its likely cause.

If a rule keeps failing to evaluate, its chip shows Failing and then Suspended. Save any change to the alert to resume it.

When something else fits

• If your worry is an agent calling the same tool again and again, the pre-defined Tool Loop alert watches for that without any rule to build.
• If you want to know when ingestion stops entirely, a Traces count rule with a threshold of less than 1 over a generous window works as a dead-man switch.
• If you want to watch call volume without being notified, the usage view on Usage and cost shows trends, but it does not trigger anything.

Related