Skip to Content

My LlamaIndex agent got worse after a release. Why?

Rius shows what changed between releases by comparing agent versions and recording each LlamaIndex step as a span, so you see whether retrieval, the model call or cost moved. Start on the agent’s page, compare the new version with the old one, then open a bad trace and read its retriever, embedding and model spans. Then add an alert so the next regression reaches you the day it ships.

How it works

The LlamaIndex integration turns each query into a nested trace of typed spans:

  • Retriever spans for retrieval steps. Their output shows what the model was given to work with.
  • Embedding spans for embedding calls.
  • LLM spans for synthesis calls, with model, token usage and messages. Rius computes the cost from the model and token counts.

The symptom tells you which span to read first:

  • Wrong or thin answers. Read the retriever span. Document ids and scores stay visible even when content capture is off, so you can see whether the new release retrieved different documents.
  • Higher cost or slower answers. Read the LLM spans. Compare input tokens, and on a trace with several model calls use the context growth strip to see where the prompt grew.
  • More failures. Read the Exceptions section on the agent page, which groups errored spans by failure type.

The agent page also compares two versions side by side on traces, failed rate, cost per trace, latency, time to first token (p50) and tokens per trace. It warns when either side has fewer than 20 traces, because a difference that small can be noise. Drift charts failed rate, cost per trace and TTFT p50 over time, so you can see the day the line moved.

If cost per trace jumped and you run glassflow-rius older than 1.1.1 with the openai extra installed, LLM calls are counted twice. Upgrade before you blame the release. With the openai extra installed, the OpenAI spans of MultiModalLLM models, embedding models and retrievers start a trace of their own, so do not expect those nested under the query.

Set it up

  1. Install the integration and initialize once. The LlamaIndex page covers it in full.
import rius rius.init(api_key="ri_...", service_name="my-agent") # build indexes and query engines exactly as before

The install line is pip install "glassflow-rius[llama-index]" llama-index.

  1. Tag every release so Rius can tell versions apart. Set these in the environment of the deployed process:
RIUS_MAIN_AGENT_VERSION=prompt-v12 # the agent definition: prompt, tools, policy RIUS_SERVICE_VERSION=2026.09.29-a41f3c # the build or deploy
  1. After the release, open Agents, then your agent, and go to Versions. Use Compare to put the old version next to the new one. Look for a higher failed rate, higher cost per trace or more tokens per trace. Check Drift for the moment it started. The trace figures count only finished traces, so give a long run time to end.

  2. Open a bad trace from the new version. The Traces page lists the version of each trace in its Version column. Use Compare runs against a good run of the same agent to see which steps changed, were added or were dropped, and where cost or latency moved. To scan one kind of step across many traces, open the Spans tab and pick Retriever or LLM in the All kinds menu.

  3. On an LLM span, open Input / Output and switch to Since previous llm call. It marks each message as added, removed or changed, so a rewritten system prompt or a dropped turn stands out.

  4. Add an alert for next time. A custom alert can watch error rate, cost, input tokens or time to first token. To measure whole runs, filter it by Service or Agent. Tokens and cost live on model-call spans, so a filter that matches only other spans measures none. The pre-defined Swallowed Error alert counts errored steps inside a run that are not tool calls, which catches a retrieval step that fails while the run still returns an answer.

What you get

  • A before and after view of your LlamaIndex agent by release: failed rate, cost per trace, latency and tokens side by side.
  • A drift line that shows the day a metric moved, so you can match it to a deploy.
  • Span-level evidence for the cause: what the retriever returned, how many tokens each model call used, and which steps differ from a good run.
  • A distinction between runs that failed and runs that recovered. Error rate counts recovered runs, while Failed traces counts only runs whose root span ended in error. A rising error rate with flat failed traces means retries or fallbacks are hiding a new problem.
  • Alerts that notify you on the next regression, instead of a user report.

For how this fits into running agents day to day, see Monitor AI agents in production.

When something else fits

  • If the symptom is answer quality with no change in errors, cost, tokens or retrieved documents, Rius shows you the spans but does not score answers. Rius has no evals, so judge quality with your own review of the Input / Output tabs.
  • If your pipeline also uses other frameworks or client libraries, check the integrations overview to see what each one records. LlamaIndex has no TypeScript integration, so LlamaIndex.TS pipelines are not traced by it.

Still stuck? Get help

Last updated on