How do I compare the cost of two models in my AI agent?

Rius compares two models when you run each as its own version of your agent: Compare shows cost per trace, tokens per trace and failed rate side by side. Rius prices every call from its model and token counts, so you send only a version label and never a cost. To check one model against another inside a single version, use cost by model on the Usage and cost page.

How it works

Rius prices each model call at ingest from the requested model name (gen_ai.request.model) and its token counts, using list prices from LiteLLM’s public catalog in US dollars. It then rolls that cost up per trace. See Track LLM and agent cost for how a call is priced.

To compare two models, give each one its own version label. The Versions section of an agent’s page reads the label from RIUS_MAIN_AGENT_VERSION (the agent definition) or RIUS_SERVICE_VERSION (the build). Compare then puts a base version next to another and shows:

• traces
• failed rate
• cost per trace
• latency and time to first token (p50)
• tokens per trace, including prompt cache, with the change between the two

Cost per trace and tokens per trace tell you what each model costs for the same kind of work. Failed rate sits in the same table, so a cheaper model that fails more often shows up in one view.

Set it up

1. Send traces from your agent as usual, with a service name or agent name set. The Agents page finds the agent from its traces, with no registration.
2. Run the agent twice, once on each model, and give each run its own version label. Use the same prompt and tools in both, so the model is the only difference:

# Deployment or run on model A
export RIUS_MAIN_AGENT_VERSION=model-a
 
# Deployment or run on model B
export RIUS_MAIN_AGENT_VERSION=model-b

3. Send similar traffic to both. Compare warns when either side has fewer than 20 traces, because a difference on so few runs can be noise.
4. Open the agent on the Agents page and choose a time window that covers both runs. In Versions, open Compare. By default it takes the two most recently introduced versions, with the older as the base. Pick your two labels if they are not those.
5. Read cost per trace and tokens per trace first. Then read failed rate and time to first token (p50) next to them. If your traces carry both an agent version and a build, switch between Agent version and Build.
6. Open Drift to see cost per trace, failed rate and time to first token over time. A gap that holds across days means more than one that appears on a single day.

If both models run inside the same version, for example a router that splits traffic, use cost by model on the Usage and cost page. It lists calls, latency, tokens and cost for each model.

You can also ask an MCP client. The agent_versions and compare_agent_versions tools expose the same comparison.

What you get

• Cost per trace and tokens per trace for each model, side by side, with the change.
• Failed rate, latency and time to first token (p50) in the same table, so you see what a cheaper model costs in reliability and speed.
• Drift charts that show whether the gap holds over time.
• Prices that follow the current catalog, with cache reads and writes at their own rates.

Two points affect how you read the numbers:

• Costs are list-price estimates. Rius does not apply batch, long-context or negotiated discounts, so your invoice can differ. Use the numbers to rank the two models, not to reconcile a bill.
• A model missing from the price catalog gets an empty cost, not zero. Averages per trace divide by every trace, so a partly priced model looks cheaper than it is. Check that both models are priced before you trust the gap.

Failed rate counts runs whose root span ended in error. It shows reliability, not answer quality. Rius does not offer scores or evals.

When something else fits

• If you need to judge answer quality, such as correctness or tone, Rius does not score outputs. Use your own evaluation process next to the cost and failed-rate comparison.
• If you need exact billed amounts, use your provider’s invoice, because Rius shows list-price estimates.

Related