Skip to Content

My Claude Agent SDK agent got worse after a release. Why?

Rius shows what changed after a release by comparing agent versions on failed rate, cost per trace, latency and tokens, then letting you open a trace to check the model. For a Claude Agent SDK agent, each task is one trace with one generation per runtime turn, so the version comparison and that generation’s model, tokens and cost are the first places to look. Tag each release with a version so the next regression is visible from the day it ships.

How it works

The Claude Agent SDK runs the Claude Code runtime in a separate process, so Rius records it with manual tracing. Each task is the root AGENT span from your observe wrapper. Inside it is one generation for the whole runtime turn, filled from the run result with the model, token counts and cost. A trace looks like this:

handle-task AGENT 5.0s └─ claude_agent_turn LLM 5.0s claude-haiku-4-5, tokens + cost

That shape tells you which symptoms Rius can explain:

  • A different model. The generation shows the requested and the response model. If a release changed either, the span’s Model section shows it.
  • More tokens or higher cost per task. A longer prompt, more history or more cache use raises tokens per trace and cost per trace.
  • Slower tasks. The root span and the generation carry the task’s duration.
  • More failures. A failed trace is a run whose root span ended in error. A trace with an errored span but a clean root reads as recovered. The runtime reports a failed turn on ResultMessage.is_error instead of raising, so raise when it is set, or those runs read as completed.
  • A blank cost. A generation recorded with a model alias such as haiku instead of a full model id has no cost. A release that changed the id can look like a cost drop when pricing is missing.

The runtime’s inner work is not in the trace. Tool calls, sub-turns and file operations inside the Claude Code process produce no spans, and the generation holds the turn’s totals only. You see that a turn got heavier or slower, not which inner step caused it.

Set it up

  1. Tag every release. Set the versions in the environment of the process that runs your agent. RIUS_MAIN_AGENT_VERSION is the agent definition (prompt, tools, policy) and RIUS_SERVICE_VERSION is the build.

    RIUS_MAIN_AGENT_VERSION=prompt-v12 # the agent definition: prompt, tools, policy RIUS_SERVICE_VERSION=2026.09.29-a41f3c # the build or deploy
  2. Compare the two versions. Open Agents, click your agent, and go to Versions. Compare puts a base version next to another and shows traces, failed rate, cost per trace, latency, time to first token (p50) and tokens per trace, with the change. It warns when either side has fewer than 20 traces, because the difference can be noise. Then read Drift for failed rate and cost per trace over time, and Exceptions to see whether the errors share one cause.

  3. Open a trace from each version. In Exploring traces, the Version column and filters narrow the list. Open a trace and select the generation. Check the response model, the input and output tokens and the cost. Use Compare runs on the trace page to put it beside another run of the same agent and see where cost or latency moved.

  4. Check the generation is set up fully. The generation needs a full model id, provider="anthropic", and usage set from the run result, including both cache counts. Without the cache fields, cache-heavy runs undercount tokens badly. The Claude Agent SDK page has the complete snippet. Use the same page’s Verify step after a release: the generation should show a full model id, tokens and cost.

  5. Add alerts for next time. Create custom alerts on cost, tokens, duration, or error rate. To alert on model spend, filter by Model, or set Span kind to LLM, because tokens and cost sit on the generation. To measure whole runs, filter by Service or Agent instead.

What you get

  • A before and after view of each release: failed rate, cost per trace, latency and tokens per trace for the two versions side by side.
  • Drift charts that show whether the change was a step on release day or a slow climb.
  • Per-trace evidence: the model that actually answered, the token counts and the cost for each task.
  • Exceptions grouped by failure type, so you can tell one cause from several.
  • Alerts on cost, tokens, duration or error rate that fire when the next release moves them.

Costs are list-price estimates in US dollars, not billed amounts. For the wider picture across agents, see Monitor AI agents in production. Other frameworks are listed under Integrations.

When something else fits

  • If the numbers look the same but the answers are worse, read the Input / Output tab of a few traces from each version. Rius does not score answer quality.
  • If the regression is in a tool call or file operation inside the Claude Code runtime, the trace cannot show it. Add your own observe spans around the steps you control.

Still stuck? Get help

Last updated on