How do I trace a RAG chatbot built with CrewAI?

Rius traces a CrewAI RAG chatbot by capturing its model calls through the openai integration and by wrapping your run, tools and retrieval step with observe. Every model call arrives with model, tokens and cost. Your wrappers add the run root, a retrieval span with the documents found, and a span for each stage. Scope each turn with a session and a user to group a conversation and the person it served.

How it works

CrewAI 1.x drives the OpenAI client natively, so the openai integration records each model call as a generation with model, tokens, cost and time to first token. If LiteLLM is installed, CrewAI uses it as a fallback, and you use the litellm integration instead. The CrewAI integration page covers this coverage rule in full, and the Integrations overview lists every supported library and framework.

Crews and tasks run inside the framework, where auto-instrumentation cannot see them, so a RAG chatbot needs a thin layer of observe wrappers on top:

• A run root. Wrap the function that calls crew.kickoff() with kind=SpanKind.AGENT. Every model call and tool call of the turn nests under it.
• A retrieval span. Open a RETRIEVER span around the search. Pass data_source_id for the index or knowledge base and top_k for how many documents you asked for. An unnamed span is called retrieval product-kb.
• A tool span for each tool. Put observe with kind=SpanKind.TOOL on the tool body. Without it, a bare observe makes a CHAIN span, which the Tool kind filter in the Spans tab never shows.
• A step for each stage. Wrap each stage function you want to see, such as query rewriting, reranking or answer drafting. A bare observe gives a CHAIN span named after the function, and kind=SpanKind.RERANKER fits a step that reorders retrieved documents.
• A session and a user. rius.session() stamps session.id and rius.user() stamps user.id on every span started in scope, including the spans from the integration.

Set it up

1. Install the SDK with the openai instrumentation next to CrewAI. Installation covers the rest of the setup.

pip install "glassflow-rius[openai]" crewai

2. Wrap the retrieval tool, the run and the turn. The names search, d.text and the ids are your own retrieval code. The @tool decorator goes outermost and observe goes on the function body.

import rius
from crewai import Agent, Crew, Task
from crewai.tools import tool
from rius import SpanKind
 
rius.init(api_key="ri_...", service_name="my-agent")
 
@tool("search_kb")
@rius.observe(kind=SpanKind.TOOL)
def search_kb(query: str) -> str:
    """Search the product knowledge base."""
    with rius.start_as_current_span(
        kind=SpanKind.RETRIEVER, data_source_id="product-kb", top_k=5
    ) as obs:
        obs.set_input(query)
        docs = search(query)
        obs.set_retrieved_documents([{"id": d.id, "score": d.score} for d in docs])
        return "\n".join(d.text for d in docs)
 
support_agent = Agent(
    role="Support specialist",
    goal="Answer questions from the knowledge base.",
    backstory="You answer product questions.",
    tools=[search_kb],
    llm="gpt-5-mini",
)
 
@rius.observe(name="answer-question", kind=SpanKind.AGENT)
def answer_question(question: str) -> str:
    task = Task(
        description=f"Answer this question: {question}",
        expected_output="A short, direct answer.",
        agent=support_agent,
    )
    return str(Crew(agents=[support_agent], tasks=[task]).kickoff())
 
def handle_turn(conversation_id: str, user_id: str, message: str) -> str:
    with rius.session(conversation_id), rius.user(user_id):
        return answer_question(message)

3. Reuse your own conversation id for the session so every turn of a chat groups together. Pass an opaque user id rather than an email, because the value is stored on every span and shown in the console. Do not set a process-wide session on a server that handles many users, since all their traces would merge into one session.

4. Set CREWAI_TRACING_ENABLED=false wherever no terminal is attached, such as CI or containers, because CrewAI asks about its own tracing on first run.

5. Run one question that calls the search tool, then open the trace in the console. You should see the root AGENT span, the LLM generations, your TOOL span and the RETRIEVER span under it.

What you get

• One trace for each chatbot turn, rooted at your answer-question span, with every model call under it showing model, tokens and a list-price cost estimate in US dollars. Rius computes the cost, so you never send one.
• A retrieval span that records which documents the search returned. set_retrieved_documents() stores identifiers and scores, not document text, so it is metadata and survives content stripping. To capture the retrieved text as content, pass it to set_output().
• Sessions that group the turns of a conversation, and a Users view that ties each person’s traces, failures and cost together. With no session scope, each trace counts as its own session. With no user scope, spans are anonymous.
• Named tool spans you can filter by kind in the Spans tab.

Know these blind spots. Crews and tasks produce no spans of their own, so wrap per-task entrypoints with observe if you need to see which task a model call belongs to. Generation spans are named ChatCompletion. Streamed calls carry token counts only when usage is enabled on the underlying OpenAI client, see the OpenAI integration note on include_usage.

When something else fits

• If LiteLLM is installed in your project, CrewAI routes through it, so enable the litellm integration instead of openai, as the CrewAI page explains.
• If your retrieval call is already made through an instrumented client library or framework, check that library’s page in Integrations before adding a manual span.
• If you need a model call that no integration records, trace it as a manual generation, as shown in Tracing your code.

Related