My Pydantic AI agent got worse after a release. Why?
Rius shows what changed between releases by comparing versions of your Pydantic AI agent on failed rate, cost per trace, latency, time to first token and tokens per trace. Tag each release with RIUS_MAIN_AGENT_VERSION, open the agent on the Agents page, and use Versions and Drift to see which number moved and when. Then open a trace from the new version: the spans Pydantic AI emits for the run, each model request and each tool call show which step changed.
How it works
Pydantic AI emits its own OpenTelemetry spans, and Rius reads them without wrappers. Each agent run is a root span named invoke_agent plus your agent’s name. Each model request is a generation named chat plus the model, with tokens and cost. Each tool call is a span named execute_tool plus the tool name. The Pydantic AI integration page covers the details.
The symptom you see usually maps to one of these places:
- Failed rate up. A failed trace is a run whose root span ended in error. A recovered trace has errored child spans but a clean root, so retries and fallbacks cost time and money without breaking the result.
- Cost per trace or tokens up. A longer system prompt, a different model, or a tool that returns more text raises the tokens on the
chatspans. - Slower runs or slower first token. More
chatcalls per run, a slower model, or a tool that takes longer. - Different behavior. Extra, missing or repeated
execute_toolspans in the tree.
Trace figures count only finished traces, so a run appears in them once every span of it has ended. Costs are list-price estimates in US dollars, computed by Rius from the model and token counts.
Set it up
- Tag every release. Set the agent definition version (prompt, tools, policy) and the build version in the environment of each deploy:
RIUS_MAIN_AGENT_VERSION=prompt-v12 # the agent definition: prompt, tools, policy
RIUS_SERVICE_VERSION=2026.09.29-a41f3c # the build or deploy- Make sure Pydantic AI sends its spans through Rius. This is the only code the integration needs, and Integrations lists every other framework:
import rius
from pydantic_ai import Agent
rius.init(api_key="ri_...", service_name="my-agent")
Agent.instrument_all(True)- Open Agents, then your agent. In Versions, use Compare to put the old version next to the new one. It shows traces, failed rate, cost per trace, latency and time to first token (p50), and tokens per trace. Rius warns when either side has fewer than 20 traces, because the difference can be noise.
- Check Drift for the failed rate, cost per trace and time to first token over time, to see the day the change began. Then read Exceptions, which groups errored spans by failure type, to see whether one cause or several explain the errors. Agents describes each section.
- Open a trace from the new version in Traces. Use the Spans tab and its All kinds menu to look at every LLM call or every tool call. Read Exploring traces for the full set of filters.
- Inside the trace, use these tools:
- The context growth strip plots input tokens for each model call, so a runaway prompt or unbounded history shows first.
- Since previous llm call on a span’s Input / Output tab shows what each step added to the prompt. This needs captured content, and what the spans carry is set by Pydantic AI’s
InstrumentationSettings(for exampleinclude_content) on top of the SDK’s privacy controls. - Compare runs sets the open trace beside another run of the same agent, to see which steps changed, were added or were dropped.
- Retry chains collapse repeated failing tool calls into one row, such as three attempts that ended recovered.
Successful Pydantic AI spans arrive with an unset status rather than OK. The trace status still reads Ok, so this is not a sign of a problem.
What you get
- A before and after comparison of two releases on failed rate, cost per trace, latency, time to first token and tokens per trace, so you know which number changed.
- A timeline of the change on the Drift charts, and a grouped list of errors on Exceptions.
- A span tree for each run, named after your agent, model and tools, so a changed tool call or an extra model request stands out.
- Prompt growth per call and per-step input diffs, for regressions caused by longer prompts.
- A way to catch it next time. Rius has four pre-defined alerts: Consistent Error Pattern, Swallowed Error, Tool Loop and Context rot. You can also write a custom alert on error rate, cost or tokens. To measure whole runs, filter by Service or Agent. To alert on model spend, filter by Model or set Span kind to LLM. Send alerts to a webhook, email or Slack.
For the wider picture across all your agents, see Monitor AI agents in production.
When something else fits
- Rius shows structure, errors, cost, tokens and timing. It does not score answer quality, run evals or manage prompts, so a drop in answer quality with unchanged numbers needs your own review of the captured inputs and outputs.
- If the version comparison shows no difference, check the model and prompt for a hidden change, and make sure both versions carried a version tag.