My AI agent got worse after a release. How do I find the cause?

Rius shows whether a release changed your agent on the agent’s page, where Versions and Drift put the new version next to the old one on failed rate, cost per trace, latency and tokens. Open the agent first, then follow its exceptions into the failed traces and read the spans that changed. The cause usually shows as an errored span, a longer prompt, a different model or an extra retry.

How it works

Rius finds your agents from their traces, so there is nothing to register. The symptom shows up in three places, from wide to narrow:

• Agents list. The Errors and Cost columns show which agent got worse. Rows with problems come first.
• Agent page. Versions compares two versions side by side on traces, failed rate, cost per trace, latency, time to first token (p50) and tokens per trace. Drift charts failed rate, cost per trace and time to first token over time, so you see the day it changed. Exceptions groups errored spans by failure type and tells you whether one cause or several are behind the errors.
• Traces. Recent and errored traces open into a waterfall of spans. Failed spans are highlighted along the path from the root to the error, and the detail panel leads with the exception and traceback.

Two numbers mean different things. Failed traces counts runs whose root span ended in error. Error rate also counts runs that hit an error and recovered, for example through a retry or a fallback. A rising error rate with steady failed traces means the agent works harder to reach the same result, which costs time and money.

For the wider picture of watching agents after a deploy, see Monitor AI agents in production.

Set it up

1. Tag your runs with a version before the next release. The comparison needs traces from both the old and the new version. Set the agent definition (prompt, tools, policy) and the build:

RIUS_MAIN_AGENT_VERSION=prompt-v12      # the agent definition: prompt, tools, policy
RIUS_SERVICE_VERSION=2026.09.29-a41f3c  # the build or deploy

2. Open Agents, scan Errors and Cost, and open the agent that is behind the others.
3. In Versions, use Compare to put the old version as the base next to the new one. Rius warns when either side has fewer than 20 traces, because a small difference can be noise. Then check Drift for the point where failed rate, cost per trace or time to first token moved.
4. Read Exceptions. One dominant failure type points at one change. Several types point at something broader, such as a model swap.
5. Check Instances. An instance that ends as gone after the release was killed before its final heartbeat, which points to a crash, an out-of-memory kill or a deploy that sent SIGTERM. A hung process that still sends heartbeats shows as running, so heartbeats do not prove progress.
6. Open Exploring traces. Use View all in Traces from the agent page, or add a filter on the version, then open a failed trace and a healthy one from the same window.
7. Read the spans that changed:
   • The Model section of an LLM span shows the requested and response model, tokens and cost. A different model or a jump in tokens after the release shows here.
   • The context growth strip above the tree plots input tokens per call. A runaway prompt or unbounded history shows up as rising bars.
   • Since previous llm call on the Input / Output tab marks each message as added, removed or changed, so a rewritten system prompt or a dropped turn stands out. It needs content capture to be on.
   • Retry chains collapse repeated calls into one row such as fetch_article · 3 attempts · recovered. A new retry loop is a common release regression.
   • Compare runs sets the open trace against another run of the same agent, side by side, to show which steps were added or dropped and where cost or latency moved.

What you get

• A verdict on the release from the day it shipped: failed rate, cost per trace, latency and tokens for the old and new version in one view.
• A short path from the symptom to a span: agent, then exception group, then failed trace, then the exact call and its exception.
• Evidence you can point to in the spans: model, token counts, context use, retries and prompt changes between calls.
• Cost figures that are list-price estimates in US dollars, which is enough to see a change in cost per trace between versions.
• A way to hand the failure on. Download a trace or span as JSON, or use Share on the trace list for a read-only, frozen view that carries summary tiles and metadata only, never span content.

When something else fits

If answers got worse but nothing errored and cost, tokens and latency are flat, Drift will not show a change. Rius does not score answers or run evals, so you read the Input / Output of the spans yourself. If you never set a version, use Drift to find the date the change began, and set the version variables before the next release.

Related