My OpenAI Agents SDK agent got worse after a release. Why?
Rius shows what changed after a release by comparing agent versions and the spans of each run, so you can see whether failures, cost or latency moved. Open the agent on the Agents page, compare the new version with the old one, then read the spans of a bad run to find the step that changed. Rius records the root run, every model call and every tool you wrap, and the pre-defined alerts catch the same pattern next time.
How it works
The OpenAI Agents SDK integration records these spans, and each one points to a different kind of regression:
- The root AGENT span is your
observewrapper around the run. It decides whether a trace counts as failed, because a failed trace is one whose root span ended in error. - An LLM generation for each model call, with model, tokens, cost and time to first token. A prompt that grew, a different model or extra calls show up here.
- A TOOL span for each function tool you wrap with
observe. A tool that now errors or is called more often shows up here. - No span for handoffs. A handoff appears only as consecutive generations that belong to different agents. Generation names are generic (
Response), so the name alone does not say which agent spoke.
Built on those spans, the Agents page answers “did the last release help or hurt?”. Versions puts two versions side by side on traces, failed rate, cost per trace, latency, time to first token (p50) and tokens per trace. Drift plots failed rate, cost per trace and time to first token over time, so you see the day the line moved. Rius warns when either side of a comparison has fewer than 20 traces, because the difference can then be noise.
Set it up
- Tag every deploy so Rius can tell versions apart. Set both variables in the agent’s environment:
RIUS_MAIN_AGENT_VERSION=prompt-v12 # the agent definition: prompt, tools, policy
RIUS_SERVICE_VERSION=2026.09.29-a41f3c # the build or deployWithout a version, the Agents page says the agent reports none and tells you which variable to set.
- Wrap each function tool with the TOOL kind, with
@function_tooloutermost, so tool problems are visible and the Tool Loop alert can see them. This is the pattern from the integration page. Setup, install andinitare covered in full there.
import rius
from agents import function_tool
from rius import SpanKind
rius.init(api_key="ri_...", service_name="my-agent") # reads RIUS_*_VERSION
def lookup(email: str) -> str:
return "active" # replace with your account lookup
@function_tool
@rius.observe(kind=SpanKind.TOOL)
def account_status(email: str) -> str:
"""Fetch a customer's account standing by email."""
return lookup(email)-
After the release, open the agent and look in this order:
- Versions, then Compare: which number moved, failed rate, cost per trace, latency or tokens per trace.
- Drift: whether the change started when the release shipped.
- Exceptions: whether the errors come from one cause or several. Compare Error rate with Failed traces. A high error rate with few failed traces means the agent recovers through retries or fallbacks, which costs time and money without breaking the result.
- Context: whether peak usage climbed toward the model’s window.
-
Open a bad trace in Exploring traces and use the tools on the trace page:
- Compare runs against a good run of the same agent shows which steps were added or dropped and where cost or latency moved.
- The Context growth strip plots input tokens per model call, so a runaway prompt shows first.
- Since previous llm call on an LLM span shows what each step added to the prompt, so a dropped turn or a rewritten system prompt stands out.
- Retry chains collapse repeated tool calls into one row such as
fetch_article · 3 attempts · recovered. - Press
]eto jump to the next failed span.
What you get
- A before and after comparison of two versions, with the day the change started on the Drift charts.
- A span-level view of what moved: model, tokens and cost on each model call, and name, status and retries on each tool.
- Alerts for next time. Rius has four pre-defined alerts: Consistent Error Pattern, Swallowed Error, Tool Loop and Context rot. You can turn them off but not delete them. Tool Loop counts only spans of kind TOOL, so a tool traced with a bare
observeis never counted. That is why step 2 sets the kind. - Custom alerts on cost, tokens, duration, time to first token and error rate. To alert on model spend, filter on Model, or set Span kind to LLM. A filter that selects only tool or agent spans measures no tokens and no cost. Use Service or Agent to measure whole runs.
- Costs are list-price estimates in US dollars, so compare versions with them, not with your invoice.
For wider monitoring habits, see Monitor AI agents in production. For other frameworks, see Integrations.
When something else fits
- If the release changed handoffs, wrap each agent’s entry with
observe, or open spans at agent boundaries with the framework’sRunHooks, because handoffs produce no span of their own. - If the answers got worse but failures, cost and latency did not move, Rius does not score outputs or run evals. You can read the prompt and completion on each span’s Input / Output tab, but judging quality needs a process of your own.
- If you set no version before the release, there is no baseline to compare. Use Drift and Compare runs on traces from before and after the deploy date.