How do I tell whether a new version of my AI agent made it worse?

Rius compares two versions of an agent side by side on traces, failed rate, cost per trace, latency and time to first token, and tokens per trace. Tag each deploy with a version, open the agent on the Agents page, and read the change between the two versions. Rius warns you when either side has fewer than 20 traces, because the difference can then be noise.

How it works

Rius reads the version from the traces your agent sends. You set it with an environment variable, and Rius groups traces by it.

• RIUS_MAIN_AGENT_VERSION is the agent definition: prompt, tools and policy.
• RIUS_SERVICE_VERSION is the build or deploy.

In Versions on the agent page, Compare puts a base version next to another. By default it picks the two most recently introduced versions, with the older one as the base. For each side it shows:

• traces
• failed rate
• cost per trace
• latency and time to first token (p50)
• tokens per trace, including prompt cache

Next to each pair you see the change. A failed trace is a run whose root span ended in error, so failed rate tells you how often whole runs break. Cost is a list-price estimate in US dollars, so compare it between versions rather than treating it as a billed amount.

If your traces carry both versions, switch between Agent version and Build. Use Agent version to judge a prompt, tool or model change, and Build to judge a code release.

Only finished traces count. A run shows up in the figures once every span of it has ended, so a long session still in progress does not skew the comparison.

Set it up

1. Set a version on each deploy. Change the value whenever you change the prompt, tools, policy or model, and whenever you ship a new build:

RIUS_MAIN_AGENT_VERSION=prompt-v12      # the agent definition: prompt, tools, policy
RIUS_SERVICE_VERSION=2026.09.29-a41f3c  # the build or deploy

2. Keep sending traces with the Rius SDK you already use. If your agent is already on the Agents page, you need nothing else. Without a version, the page says “This agent reports no version” and tells you which variable to set.
3. Let both versions handle real traffic. The comparison needs at least 20 traces on each side to be free of the warning, so give the new version time before you judge it.
4. Open Agents, click your agent, and scroll to Versions. Check that Compare shows the base and the version you shipped. Pick different ones if the defaults are not the pair you want.
5. Read the table. A new version is worse if its failed rate or cost per trace rose, or if latency, time to first token or tokens per trace went up without a result you wanted in return.
6. Open Drift to see when the change started. Three charts show time to first token (p50), cost per trace and failed rate over time. There is one point per minute, hour or UTC day depending on the window, and each trace counts in the bucket where it started. A step on the day you shipped points at the release. A slow climb points at something else.
7. Use the time window at the top right (15m, 1h, 1d, 7d, 30d) so that the window covers traffic from both versions.

What you get

• A side-by-side view of two versions with the change on each measure, so you see whether the release raised the failed rate or the cost per trace from the day it shipped.
• A warning when a side has fewer than 20 traces, so you do not act on noise.
• Drift charts that show when a metric moved, not only that it differs.
• A way to separate a prompt or model change from a build change, using the Agent version and Build switch.
• A cross-check from the summary tiles. Failed traces counts runs whose root span ended in error. Error rate also counts runs that recovered, for example with a retry. If the error rate rose but failed traces did not, the new version recovers more often, which costs time and money without breaking the result.
• The same data for an agent that queries it: the MCP tools agent_versions and compare_agent_versions return version lists and comparisons.

For the wider routine around releases, alerts and cost, see Monitor AI agents in production.

When something else fits

The comparison tells you that a version changed failed rate, cost, latency or tokens, not whether its answers are better. Rius does not score answers or run evals, so judge quality with your own checks. If traffic is too low to reach 20 traces per side, widen the time window or wait before you decide.

Related