How do I monitor a RAG chatbot in production?

Rius makes a RAG chatbot run readable by tracing it as one root span, with a span for each model call and a span for each retrieval or tool step. Once it is live, you watch the Traces waterfall, the Agents page and three pre-defined alerts. Rius computes cost and tokens for each call from the spans, so you only wrap your own functions. The full setup is in Monitor AI agents in production.

How it works

Each chatbot turn becomes one trace with three kinds of span:

• The run. The function that handles a user message is the root. Wrap it with observe and pass kind=SpanKind.AGENT (kind: SpanKind.AGENT in TypeScript), so the trace is labeled as an agent run. A bare observe records a generic CHAIN span.
• Each LLM call. With a supported library installed and init() called, every call becomes a generation span with its model, token counts and cost. You add no code for this.
• Each retrieval step. Open a span with kind=SpanKind.RETRIEVER around the search, with data_source_id and top_k. Each tool the model calls, such as an order lookup, gets observe with kind=SpanKind.TOOL. The retrieval span sits in the waterfall between the root and the generation that uses the retrieved text.

The resulting waterfall shows the order of a turn: retrieval first, then generation, with timing and errors on each span. A failing step is visible as an errored span, not a vague failed answer.

A TOOL span is named after the tool, which defaults to the function’s name, so lookup_order shows as execute_tool lookup_order. The AGENT root is named after the agent, as invoke_agent followed by the agent name.

Set it up

1. Install the Rius extra for the LLM library your chatbot uses and call init() once at startup. The Monitor AI agents in production guide covers installation in full, for Python and TypeScript.

   import rius
    
   client = rius.init(
       api_key="ri_...",
       service_name="rag-chatbot",
   )

2. Wrap the handler that answers one user message with observe, kind AGENT.

3. Trace each retrieval as a RETRIEVER span and each tool the model calls with observe, kind TOOL. Tool spans matter for monitoring: the Tool Loop alert only works on TOOL spans.

4. Group the turns of one conversation by setting session.id, with rius.session in Python or withSession in TypeScript. Set user.id with rius.user or withUser to see each end user on the Users page. Prefer an opaque user id to an email.

5. Tag each release so you can compare versions later. Set the version of the prompt, tools and retrieval settings as the agent version, and the deploy as the service version:

   export RIUS_MAIN_AGENT_VERSION=prompt-v12
   export RIUS_SERVICE_VERSION=2026.09.29-a41f3c

6. Create a distribution channel (Slack, email or a webhook) and attach it to each alert from its Delivers to section. Without a channel, a firing is only a row on the Alerts page.

What you get

Views to watch once the chatbot is live

• Traces. Open any slow or wrong turn and read the waterfall from the root down to the retrieval step and the generation. Token counts, cost and errors are on the spans.
• Home. Volume, cost by model, tokens and error rate per period. Costs are list-price estimates in US dollars, not billed amounts.
• Agents. Open the Agents page and your chatbot’s entry. Check Errors and Cost for the trend, Drift for a rising failed rate or cost per trace, and Exceptions to see whether errors share one cause, such as the retrieval step failing the same way.
• Versions. Compare two versions on failed rate, cost per trace, latency and tokens per trace. Rius warns when a side has fewer than 20 traces. This shows whether a new prompt or model helped.
• Context. Peak usage shows how close the largest call came to the model’s context window. Check it if retrieved passages make your prompts long.
• Instances. Shows whether each process is running, ready, stale, gone or stopped.

Alerts that fit a RAG chatbot

• Consistent Error Pattern fires when the same error recurs across runs, for example a retrieval backend failing the same way on many turns.
• Swallowed Error fires when a run reports success while a step inside it failed. That covers a chatbot that answers even though retrieval errored, because a retrieval span is not a tool call.
• Tool Loop fires when the same tool is called with identical arguments three or more times in a row within one trace, such as a lookup tool repeating the same query. It is on in every workspace.

Add custom alerts for your own thresholds, for example on cost, tokens, duration, time to first token, context usage or error rate.

When something else fits

• Rius traces, prices and alerts on runs. It does not score answer quality or offer evals, datasets or experiments, so judging whether answers are grounded needs another tool.
• Heartbeats show that a process is alive, not that it is making progress. A hung chatbot process that still sends heartbeats shows as running, so also alert on duration.

Related