Skip to Content

My AutoGen agent got worse after a release. Why?

Rius shows what changed between the release and the one before it by comparing the two versions of your AutoGen agent on failed rate, cost per trace, latency and tokens per trace. Then you open a slow, costly or failed trace and read its spans: the framework’s invoke_agent and execute_tool spans and the ChatCompletion generations with model, tokens and cost. Tag each release with a version so the comparison exists, and add a custom alert so the next regression reaches you.

How it works

The AutoGen integration records the run in two layers. AutoGen emits create_agent, invoke_agent and execute_tool spans itself, and the openai integration adds a ChatCompletion generation for each model call, with tokens and cost. Your observe wrapper on the entrypoint is the root AGENT span, so one unit of work is one trace.

That gives you three kinds of evidence after a release:

  • Version-level numbers. The agent page on Agents compares two versions side by side: traces, failed rate, cost per trace, latency and time to first token (p50), and tokens per trace.
  • Trace-level structure. Which agent or tool steps were added, dropped or repeated.
  • Call-level detail. Each generation’s model, input and output tokens, and cost. Costs are list-price estimates in US dollars, not billed amounts.

See Integrations for the other frameworks Rius covers, and Monitor AI agents in production for the wider workflow.

Set it up

  1. Tag each release. Rius reads the version from the environment. RIUS_MAIN_AGENT_VERSION is the agent definition (prompt, tools, policy) and RIUS_SERVICE_VERSION is the build.

    RIUS_MAIN_AGENT_VERSION=prompt-v12 # the agent definition: prompt, tools, policy RIUS_SERVICE_VERSION=2026.09.29-a41f3c # the build or deploy

    Without a version, the agent page says it reports none and tells you which variable to set.

  2. Compare the two versions. Open your agent on Agents, go to Versions and use Compare. By default it puts the two most recently introduced versions side by side, the older as the base. Read which number moved. A warning appears when either side has fewer than 20 traces, because a small difference can be noise. If traces carry both values, switch between Agent version and Build.

  3. Check Drift. The Drift charts plot TTFT p50, cost per trace and failed rate over time, so you see whether the change starts on the day the release shipped or earlier. Exceptions groups errored spans by failure type, which tells you whether one cause or several are behind the errors.

  4. Open a trace from each version. On Exploring traces, read the Version column to find a trace from each version, or start from the agent’s Versions section on the Agents page, and open a trace. Use Compare runs against another run of the same agent to see which steps changed, which were added or dropped, and where cost or latency moved. Then read the spans:

    • Several ChatCompletion generations where the old version had one means the agent loops more.
    • The Context growth strip plots input tokens per model call, so a prompt that grows with each step stands out.
    • On a generation, Since previous llm call in the Input / Output tab marks what each step added, changed or removed, so a rewritten system prompt is visible.
    • Repeated execute_tool spans with the same name show a tool being called again and again. A retry that eventually succeeds collapses into one row such as fetch_article · 3 attempts · recovered.
  5. Read the status carefully. AutoGen’s own spans arrive with status UNSET on success, and the trace status still reads Ok. A trace badged recovered had errored child spans while the root finished cleanly, which fits an agent that now retries more.

  6. Alert on it next time. Create a custom alert on a rule such as cost, tokens, duration or error rate. To alert on model spend, filter by Model or set Span kind to LLM, because tokens and cost live on model-call spans. To measure whole runs, filter by Service or Agent. Alerts can go to webhook, email or Slack. The pre-defined alerts also watch for repeated errors, swallowed errors, tool loops and context rot.

What you get

  • A before-and-after table for the release: failed rate, cost per trace, latency, time to first token and tokens per trace.
  • A drift chart that shows when the change began.
  • The step-level cause in the trace: extra model calls, a growing prompt, a repeated tool call, or errors that the agent now recovers from.
  • Failed traces separated from recovered ones, so you know whether the result broke or the run just got slower and more expensive.
  • A custom alert on cost, tokens, duration or error rate, so the next regression reaches you through your chosen channel. If an alert fires, you can also run Rius’s optional root-cause analysis on it.

When something else fits

  • If the agent got worse but no version was tagged, you can still filter by time range and compare runs, but there is no version comparison until you set RIUS_MAIN_AGENT_VERSION or RIUS_SERVICE_VERSION.
  • Do not wrap AutoGen tool functions with observe to get more detail. The wrapper splits off as its own single-span trace, and the framework’s execute_tool spans already cover tools.

Still stuck? Get help

Last updated on