How do I trace a RAG chatbot built with LangChain?

Rius traces a LangChain RAG chatbot once you install the langchain extra and call init: chains, retrievers, tools and model calls arrive as a nested trace that mirrors your graph. On top of that, you add a root span for each chatbot turn, a span for any retrieval or rerank step LangChain does not run for you, and a session and user id. Together these give you one trace per question, grouped by conversation and by person.

How it works

The LangChain integration hooks LangChain’s callback system, so anything that runs through it is traced, including LCEL chains, agents and LangGraph nodes. Each step becomes one span, typed by what it is:

• Chains are CHAIN spans.
• Retrievers are RETRIEVER spans.
• Tool calls are TOOL spans.
• Model calls are LLM spans with model, token usage and messages. Rius computes cost from these on its side.

What you add yourself, with the helpers on Tracing your code:

• A run root. Wrap the function that handles one user message with observe, so the retriever, the prompt chain and the model call share one parent and one trace per turn.
• A step per stage. Wrap your own stages, such as query rewriting or a rerank step, with observe. SpanKind.RERANKER is meant for a step that reorders retrieved documents by relevance.
• Your own retrieval. If you query a vector store outside a LangChain retriever, open a RETRIEVER span with data_source_id (the index or knowledge base) and top_k. set_retrieved_documents() records document ids and scores as metadata, not text. Put retrieved text in set_output() if you want it captured as content.
• Session and user. The session id is your conversation id, so every turn of a chat lands together. The user id is the person the request served.

If you also install a client extra such as openai, LangChain’s chat model span wraps the client span, so you get both framework context and client detail. Each chat model call is counted once in tokens and cost, and the LangChain span still shows the run’s total. Text-completion models (BaseLLM subclasses, such as langchain_openai.OpenAI) are still counted twice in that setup.

Set it up

1. Install the instrumentation and start the SDK once at startup. The extra installs the instrumentation, not LangChain itself. Full setup is on Integrations.

pip install "glassflow-rius[langchain]" langchain

import rius
 
rius.init(api_key="ri_...", service_name="my-agent")
# build and run your chains exactly as before

In TypeScript, run npm install @arizeai/openinference-instrumentation-langchain, then call init from @glassflow-ai/rius and await ready on the client it returns before you build your chains, because integrations attach asynchronously. @langchain/core is also required, and a LangChain.js app already has it.

2. Wrap the turn handler and your own stages with observe. The names below are yours.

import rius
from rius import SpanKind
 
@rius.observe
def answer_question(query: str) -> str:
    ...  # run your retriever and chain here
 
@rius.observe(name="rerank", kind=SpanKind.RERANKER, capture_output=False)
def rerank(docs: list[str]) -> list[str]:
    ...

3. If retrieval happens outside LangChain, trace it as a retriever span:

with rius.start_as_current_span(
    kind=SpanKind.RETRIEVER, data_source_id="product-kb", top_k=5
) as obs:
    obs.set_input(query)
    docs = search(query)
    obs.set_retrieved_documents([{"id": d.id, "score": d.score} for d in docs])

4. Scope the session and the user around each request. Pass your own conversation id and user id, and prefer an opaque user id to an email.

with rius.session(conversation_id):
    with rius.user(request.user_id):
        answer_question(message)

In TypeScript, use withSession and withUser. Do not set a process-wide session in a server that handles many users, because every user’s traces would merge into one session.

5. Send one question and open the trace in the console. You should see your root span, with the chain’s spans nested under it and the model call carrying model, tokens and cost.

What you get

• One trace per chatbot turn, with the retriever, chain steps, tools and model call nested under your root span.
• Retrieval spans that show which data source was searched, how many documents were requested, and the ids and scores that came back.
• Token counts and a list-price cost estimate in US dollars on each model call.
• Turns grouped by session.id, so a whole conversation reads as one session. With no session scope, each trace counts as its own session.
• Failures, cost and time to first token per user.id. Spans outside a user scope count as anonymous.

When something else fits

• If your model calls do not go through LangChain, trace them with manual generations instead.
• Some LangChain streams carry no time to first token. Trace those calls as a manual generation if you need it.
• Rius traces, prices and alerts on runs. It does not offer scores, evals or datasets for judging answer quality.

Related