What should I trace in a RAG chatbot?

Rius gives a RAG chatbot one trace per conversation turn: a root span for the run, a span for retrieval, a generation for each model call, and spans for any tools. You set a session id on the conversation and a user id on the request, so every span carries both. The result is a waterfall where you can see whether a bad answer came from retrieval, the prompt or the model.

How it works

A trace is one run of your agent, and for a chatbot that is one turn. It is a tree of spans, and each span has a kind that tells Rius what it is. The kinds that fit a RAG chatbot:

• Root span, kind AGENT. One per turn. Its duration is the latency the user feels, and a failed root marks the turn as failed. Wrap your turn handler with observe and kind=SpanKind.AGENT.
• Retrieval span, kind RETRIEVER. One per search. Set data_source_id to the index or knowledge base you query and top_k to the number of documents you ask for. Call set_retrieved_documents() with each document’s id and score. This records identifiers and scores, not text, so it stays visible even when content capture is off. Put retrieved text in set_output() only if you want it captured as content.
• Reranking span, kind RERANKER. Add it if you reorder retrieved documents by relevance, so you can see what the reranker changed.
• Embedding span, kind EMBEDDING. Add it if you embed the question in your own code. Its tokens and cost count toward your totals.
• Generation, kind LLM. One for each model call that writes an answer. It records the model, provider, token usage, finish reasons and, optionally, the messages. This is what Rius uses to compute cost and model analytics.
• Tool spans, kind TOOL. One per tool the model calls, such as an order lookup. Pass tool_name, and tool_call_id to join the span back to the generation that asked for it.

Spans nest through parent context, so the retrieval and the generation appear under the root in the order they ran. Two identifiers sit across all of them. session.id ties the turns of one conversation together, and user.id ties together everything one person did across conversations. Read more about both in Concepts.

Set it up

1. Initialize the SDK once at startup. The full setup is in Tracing your code, which also covers the TypeScript forms of everything below.
2. Wrap the turn handler as the root span, open a retrieval span around the search, and open a generation around the model call. If the openai extra is installed, leave openai out of rius.init(instruments=[...]), or the call is counted twice.
3. Scope the session and the user around each turn, outside the root span, so every span inside inherits both.

import rius
from rius import SpanKind
 
rius.init(api_key="ri_...", service_name="docs-chatbot")
 
@rius.observe(kind=SpanKind.AGENT)
def answer(question: str) -> str:
    with rius.start_as_current_span(
        kind=SpanKind.RETRIEVER, data_source_id="product-kb", top_k=5
    ) as obs:
        obs.set_input(question)
        docs = search(question)
        obs.set_retrieved_documents([{"id": d.id, "score": d.score} for d in docs])
 
    messages = build_prompt(question, docs)
    with rius.start_as_current_generation(model="gpt-4o", provider="openai") as gen:
        gen.set_input(messages)
        response = client.chat.completions.create(model="gpt-4o", messages=messages)
        gen.set_output(response.choices[0].message.content)
        gen.set_usage(
            input_tokens=response.usage.prompt_tokens,
            output_tokens=response.usage.completion_tokens,
        )
    return response.choices[0].message.content
 
def handle_turn(conversation_id: str, user_id: str, message: str) -> str:
    with rius.session(conversation_id), rius.user(user_id):
        return answer(message)

4. Use your own conversation id as the session id, so a conversation that outlives a deploy or moves between replicas stays one session. Do not set a process-wide session id in a server that handles many users, because every user’s traces would merge into one session.
5. Pass an opaque user id rather than an email. The value is stored on every span and shown in the console.
6. If you stream answers, call record_first_token() on the generation for each chunk. Rius derives time to first token from it, which is the latency a chat user notices. If you call Claude, pass provider="anthropic".

If you use auto-instrumentation for your model client, the generations are created for you, and your own spans for the root and the retrieval still nest around them.

What you get

• A waterfall for each turn that separates retrieval time from generation time, so you can see which step made a slow answer slow.
• The retrieved document ids and scores on every retrieval span, so you can check whether the right documents came back for a bad answer.
• Cost per model call, computed by Rius from the model and the token counts you send. You send token usage and never a cost value. Costs are list-price estimates in US dollars.
• Sessions that show a whole conversation turn by turn, and a user view that follows one person across conversations.
• Errors recorded on the failing span with its error type, with the path from the root to that span highlighted.

For the wider picture of watching an agent in production, see Monitor AI agents in production.

When something else fits

• If your chatbot is one model call with no retrieval or tools, a single generation under a root span is enough, and the extra span kinds add nothing.
• If you want to score answers, run evals or manage prompts, Rius does not offer those. It traces, prices and alerts on agent runs.

Related