My LlamaIndex agent fails only for some users. Why?

Rius shows whether your LlamaIndex failures cluster on a few end users, and the spans for retrieval, embedding and model calls show which step broke for them. Start on the Users page, sort by failed traces, then open one failing trace. Compare it with a run that worked to find what those users have in common.

How it works

With the Python integration on, each LlamaIndex query becomes a nested trace of typed spans:

• Retrieval steps are RETRIEVER spans.
• Embedding calls are EMBEDDING spans.
• Synthesis model calls are LLM spans with the model, token usage and messages. Rius computes the cost.

The retriever span’s output shows what the model was given to work with, which is where a failure that depends on the user’s data usually starts.

Rius attributes a trace to an end user through user.id. A trace with no end user on any span counts as anonymous. The Users page lists, for each user, their traces, failed traces and cost in the window. A user’s page adds an error rate tile and their recent and failed traces. Failed traces on that page are those whose root span ended in error. The error rate also counts runs that recovered, where a child span failed but the run completed.

A failure that hits only some users is often tied to what those users bring: their documents, the size of their data, their language or their account settings. The Users page puts that pattern in one list.

Set it up

1. Turn on the LlamaIndex integration. This is the shared setup, covered in full on the LlamaIndex integration page and in Integrations. The integration is Python only.

   pip install "glassflow-rius[llama-index]" llama-index

   import rius
    
   rius.init(api_key="ri_...", service_name="my-agent")
   # build indexes and query engines exactly as before

2. Run each request inside a user scope with rius.user, so every span of the request carries user.id. Prefer an opaque user ID to an email. Then check the line above the Users list, such as “400 of 471 traces carry an end user”. A large anonymous row means some code paths run outside a user scope, and those failures cannot be tied to anyone.

3. Open Users and sort by Failed. If a handful of users account for most failures, the problem is specific to them. Open one and compare their Error rate and cost per trace with the agent’s own numbers on the Agents page.

4. Open a trace from Failed traces on that user’s page. In the waterfall, failed spans are highlighted along the path from the root to the error, and the detail panel of an error span leads with the exception and traceback. A trace badged recovered had child spans that errored while the run finished clean, so look for retries or fallbacks. The keys ]e and [e jump between failed spans.

5. Read the step that failed. In the Spans tab, the All kinds menu filters to Retriever, LLM or Embedding. Check the retriever span’s output first. An empty or off-topic result for these users points at their data or index. If the failure is in the LLM span, compare Input / Output with Since previous llm call. This marks messages as added, removed or changed. A rewritten system prompt or a very long history stands out. The context growth strip above the tree plots input tokens per call, so a prompt that grows past the model’s window shows up there.

6. Compare the failing run with a successful run of the same agent using Compare runs on the trace page. It shows which steps changed, were added or were dropped. To list one user’s traces, click their cell in the User column of the trace list, which filters the list to them. Use the Share button if you want to hand the filtered failures to a teammate. A shared view carries metadata only, never span content.

What you get

• A list of the users who fail most, with each user’s error rate and cost next to the agent’s average.
• The failing step for those users: retrieval, embedding or the model call, with the exception and traceback on the error span.
• A side-by-side comparison of a failed run and a good one, so the difference in steps, tokens or prompt is visible.
• A routine for next time. Check the Users page sorted by Failed whenever failures rise. Open that user’s Failed traces. Custom alert rules can filter by service, agent, step, model, span kind, operation, session or instance, but not by user, so the Users page is where per-user clustering shows. An alert on error rate for the agent tells you when to look. See Monitor AI agents in production for the wider routine.

Retrieved document content and messages are content attributes. If your privacy controls strip them, the Input / Output tab says so, while document ids and scores stay visible. Timings, tokens and cost still work.

When something else fits

• If the failing users are all in the anonymous row, fix the user scope first, because Rius cannot group traces without user.id.
• If your agent runs on LlamaIndex.TS, there is no LlamaIndex integration in TypeScript. The OpenAI integration still traces provider calls, and manual generations cover the pipeline around them.

Related