My AutoGen agent fails only for some users. Why?
Rius shows which end users fail and which step fails for them, by grouping AutoGen traces by user and recording each agent, model call and tool call as a span. Sort the Users page by failures, open a failed trace, and read the waterfall to see where the run broke. Then compare those traces with a healthy user’s to find what the failing users share.
How it works
AutoGen emits its own spans for agent creation, agent runs and tool calls. The openai integration adds a span for each model call with tokens and cost. A run that calls a tool looks like this in the trace:
handle-ticket AGENT 2.8s
├─ create_agent support_agent 0ms
└─ invoke_agent support_agent 2.8s
├─ ChatCompletion LLM 2.6s
└─ execute_tool account_status 0msThat tree shows which layer fails for the affected users: a model call, a tool call or the agent step around them.
Rius derives trace status from the spans. A trace is failed when at least one span has an OpenTelemetry ERROR status. If the root span finished cleanly while a child errored, the trace is badged recovered, so a retry or fallback that hides a problem from the user is still counted. AutoGen’s own spans arrive with an unset status on success, and the trace still reads Ok.
To split failures by person, Rius reads user.id from the spans. You set it with rius.user in Python. A trace with no user on any span counts as anonymous, and a large anonymous row means some code paths run outside a user scope.
For the wider picture across agents, see Monitor AI agents in production.
Set it up
-
Attribute every request to a user. Run each request inside a user scope with
rius.user, and reuse your conversation id withrius.session(). Check it with a test ID such astest-user-1: open Users, type the ID, click Find, and confirm the row shows one trace and your agent. Without this step the failures sit in the anonymous row and cannot be split by person. -
Trace AutoGen with one wrapper on the entrypoint. The framework’s spans and the
openaiintegration cover the rest. The full setup is on the AutoGen integration page, and the other integrations are listed under Integrations.import asyncio import rius from autogen_agentchat.agents import AssistantAgent from autogen_ext.models.openai import OpenAIChatCompletionClient from rius import SpanKind rius.init(api_key="ri_...", service_name="my-agent") def lookup(email: str) -> str: return "active" # replace with your account lookup def account_status(email: str) -> str: """Fetch a customer's account standing by email.""" return lookup(email) @rius.observe(name="handle-ticket", kind=SpanKind.AGENT) async def handle_ticket(ticket: str) -> str: agent = AssistantAgent( name="support_agent", model_client=OpenAIChatCompletionClient(model="gpt-5-mini"), tools=[account_status], system_message="Answer support questions; check accounts with the tool.", ) result = await agent.run(task=ticket) return str(result.messages[-1].content) async def main() -> None: # Enter the user scope before the root span starts, so every span carries user.id with rius.user("user_8412"), rius.session("ticket-1093"): print(await handle_ticket("Why is my account on hold?")) asyncio.run(main())Do not wrap AutoGen tool functions with
observe. The framework runs tools in a detached context, so the wrapper’s span splits off as its own single-span trace. The framework’sexecute_toolspans already cover tools. -
Find who fails. On the Users page, sort by Failed. If a handful of users account for most failures, look at what they share, such as account settings, data size or language, before you change anything for everyone. Open one user and compare their Error rate and cost per trace with the agent’s own numbers on the Agents page.
-
Find where it fails. Open a trace from the user’s Failed traces list. In the waterfall, failed spans are highlighted along the path from the root to the error, and the detail panel leads with the exception and traceback. Check whether the error sits on the
ChatCompletionspan or theexecute_toolspan. In Exploring traces, the Spans tab filtered by status lists every failing span at once. -
Compare with a healthy run. Use Compare runs on the open trace against another run of the same agent to see which steps changed. On a model call’s Input / Output tab, Since previous llm call shows what each step added to the prompt, and the context growth strip shows input tokens per call, which is where a long history shows up first.
What you get
- A list of users ranked by failed traces, so you can tell whether the problem is one account, a small group or everyone.
- For each failing user, the exact span that errored, with the exception and traceback for that run.
- A way to see retries: back-to-back attempts of the same step collapse into one row, such as
3 attempts · recovered, so a hidden retry loop reads as one event. - Inputs and outputs on model calls, when the spans carry content, so you can set a failing user’s prompt beside a healthy one.
- A saved search or a copied link to the same filtered view. Filters live in the URL, and the filter builder accepts the
user_idfield, so you can hand a teammate the failing user’s traces. For a link to someone outside the workspace, Share creates a read-only view with metadata only, never span content. - A way to catch it earlier next time. A custom alert on error rate, filtered by service or agent so it measures whole runs, fires when failures rise. The Swallowed Error alert catches errored child steps that are not tool calls, such as a model call that fails while the run still completes. The Users page, sorted by Failed, then tells you whether the new failures belong to a few people.
When something else fits
- Spans show where a run broke and what differed for the failing user, but the fix to your tool, prompt or data happens in your own code.
- Custom alert filters cover service, agent, step, model, span kind, operation, session and instance, so use the Users page to split by person after an alert fires.