How do I alert on LLM cost per agent?

Yes, Rius lets you alert on LLM cost for one agent: create a custom alert, set the Agent filter to that agent, and put a threshold on the Cost metric. Cost is measured per trace, so the rule watches what a run of that agent costs, for example p95(Cost) > $2.00 over one hour. A total-spend budget for an agent is not an alert metric, and the Agents page covers that need.

How it works

A custom alert is made of filters, one or more rules and a rolling window. Filters narrow which spans feed the rules before anything is aggregated, and Agent is one of the filter fields. The others are service, step, model, span kind, operation, session and instance.

• Agent filter. It matches the agent identity Rius derives, as each span carries it. It selects every span that carries that agent value, not only root spans. Values are suggested from your workspace’s live data, and the filter offers =, ≠, in and not_in.
• Cost metric. Cost is a numeric metric in US dollars with avg, min, max, p50, p95 and p99 aggregations. It is measured per trace, so avg(Cost) is the average cost of a run in the window, not the total spent in it.
• Window. Any value from 5 minutes to 24 hours.
• Agent name. Rius takes it from your resource attributes, and falls back to the service name. If you never set an agent name, the Agent and Service filters point at the same name.

Costs are list-price estimates, so the threshold is compared against an estimate, not your invoice.

Set it up

1. Give the agent a name of its own, so the Agent filter has a clear value to pick. The Agents page lists every way Rius derives the name. The simplest is the environment variable:

   export RIUS_AGENT_NAME=checkout-agent

2. Open the custom alert builder and add the filter Agent = your agent. Then add a rule on Cost and set the window. This is the whole alert:

   FILTERS   Agent = checkout-agent
   RULES     p95(Cost) > $2.00
   WINDOW    1 hour

3. Pick the aggregation for what you want to catch. Use p95 or max to catch a few expensive runs, such as a prompt that grew or an agent stuck retrying. Use avg to catch a general drift upward.

4. Create one alert per agent if each needs its own threshold. The in operator puts several agents under one rule, but they then share one threshold.

5. Check that your models are priced. A span whose model has no known price carries no cost, and a rule over mostly unpriced models can stay quiet. Firings show the share of the window that was priced, so you can tell a real reading from a partly priced one. Cost by model on Usage and cost shows which models are priced, and Track LLM and agent cost explains how a call is priced.

What you get

• An alert that fires when the cost of a run of one agent crosses your threshold, with the observed value recorded against it on the firing.
• The coverage share on each firing, so a high reading is not confused with a partly priced one.
• Separate thresholds for separate agents, when you create one alert per agent.
• Cost per agent, without an alert, on the Agents page: total cost and average cost per trace for each agent, cost per trace over time in Drift, and cost per trace side by side when you compare two versions.

When something else fits

• If you want an alert on the total dollars an agent spends in a day or a month, a custom alert cannot do it. Cost is measured per trace, and a total-spend or budget threshold is not an alert metric. Watch the Cost tile on the agent’s page and the workspace totals on Usage and cost instead.
• If you prefer to ask in plain language, the Rius MCP server’s agent_traces_summary tool returns the cost summary for one agent.

Related