Is my AI agent on the right model?

Rius shows whether your agent is on the right model by putting its cost, time to first token and peak context usage side by side on the agent page. To test a different model, tag the run with a new version and compare it with the old one in Versions. The comparison shows failed rate, cost per trace, latency and tokens per trace for both.

How it works

Open your agent from the Agents page and read three things together over the time window you pick (15m, 1h, 1d, 7d or 30d). For more on reading traces across agents, see Monitor AI agents in production.

• Cost is the total and the average per trace. It is a list-price estimate in US dollars, not a billed amount.
• TTFT p50 is the median time to first token over your streaming model calls. The tile also shows how many calls were measured, such as 42 of 55. It shows a dash for an agent that never streams.
• Peak usage in Context is how much of the model’s context window the largest call used.

How to read them:

• An agent that uses a small share of its context window, starts answering slowly and costs a lot is a candidate for a smaller, faster model.
• An agent close to 100% peak usage needs a larger window, or less history in each prompt.

The Context section also breaks the prompt into buckets such as system prompt, tool definitions, tool calls and results, and history. You can see what fills the window before you decide to change the model.

To test a change, treat the new model as a new version of the agent. Rius reads the version from RIUS_MAIN_AGENT_VERSION (the agent definition) or RIUS_SERVICE_VERSION (the build). Versions lists each version that ran in the window. Compare puts a base version next to another and shows the change.

Set it up

1. Make sure your agent already sends traces and shows on the Agents page. The Agents page covers the setup in full, including the heartbeat settings.
2. Tag the run you have today with a version, then change the model and tag the new run with a different value. Set the variables in the environment of each deployment:

RIUS_MAIN_AGENT_VERSION=prompt-v12      # the agent definition: prompt, tools, policy
RIUS_SERVICE_VERSION=2026.09.29-a41f3c  # the build or deploy

3. Change RIUS_MAIN_AGENT_VERSION when you swap the model, for example from prompt-v12 to prompt-v13. Change RIUS_SERVICE_VERSION if you track the swap as a new build instead.
4. Run both versions on real traffic until each has at least 20 traces. Rius warns when either side has fewer than 20, because the difference can be noise.
5. Open the agent, go to Versions and use Compare. By default it compares the two most recently introduced versions, with the older one as the base. Pick other versions if you need to. If traces carry both an agent version and a build, switch between Agent version and Build.
6. Read the side-by-side numbers: traces, failed rate, cost per trace, latency and time to first token (p50), and tokens per trace. Tokens include prompt cache.

What you get

• A view of cost, time to first token and context usage for one agent, so you can tell whether the model is oversized, undersized or fine.
• A side-by-side comparison of the old and new model as two versions, with the change in failed rate, cost per trace, latency and tokens per trace.
• Drift charts for TTFT p50, cost per trace and failed rate. They show from the day a new version ships whether the swap moved anything. Each chart has one point per minute, hour or UTC day, depending on the window.
• Exceptions, which groups errored spans by failure type, so you can see whether a cheaper model fails for one reason or several.
• A clear read on errors. Compare Error rate with Failed traces. A high error rate with few failed traces means the agent recovers, for example with retries or fallbacks, which costs time and money without breaking the result.

If Rius does not know a model’s price, the cost is empty rather than zero. Check that before you trust a cost comparison.

When something else fits

Rius tells you what a model costs, how fast it answers and how often runs fail. It does not score the quality of answers, so judge those with your own checks. If your agent never streams, TTFT shows a dash, so use cost per trace, latency and tokens per trace to compare.

Related