How do I trace a RAG chatbot built with the OpenAI Agents SDK?

Rius traces a RAG chatbot built with the OpenAI Agents SDK by capturing every model call automatically and adding a few observe wrappers for the run root, retrieval and tools. Each model call arrives with model, tokens and cost. The run root and the retrieval step make the trace read as one question answered in stages. Wrap each conversation in a session and each request in a user so turns group together.

How it works

The OpenAI Agents SDK drives the OpenAI client underneath, so the openai integration records every model call without framework-specific code. This integration is Python only. In TypeScript, trace those calls with the manual generations on the tracing page.

The framework keeps the agent loop, tool runs and handoffs to itself, so a RAG chatbot needs a thin layer of your own spans:

• A run root. Wrap your entrypoint with observe and the AGENT kind. Every model call and tool call of the answer nests under it.
• A retrieval span. Open a span with the RETRIEVER kind around your vector search. Pass data_source_id (the index or knowledge base) and top_k, and Rius names the span retrieval product-kb for you.
• A rerank step, if you have one. Wrap it with observe and the RERANKER kind.
• Tool spans. Wrap the body of each function tool with observe and the TOOL kind. A bare observe makes a CHAIN span, which the Tool kind filter in the Spans tab and the Tool Loop alert never see.

set_retrieved_documents() records document ids and scores, not text. Pass retrieved text to set_output() if you want it captured as content. See Tracing your code for every option.

The framework also uploads its own traces to the OpenAI platform. Call set_tracing_disabled(True) if Rius should be your only telemetry path.

Set it up

1. Install the SDK with the OpenAI extra and the framework. Full setup is on the OpenAI Agents SDK page.

pip install "glassflow-rius[openai]" openai-agents

2. Initialize Rius, turn off the framework’s own tracing, and define the agent. Use underscores in agent names, because hyphens are rewritten in handoff tool names and the framework warns on every run.

3. Put the retrieval span, the run root and the session and user scopes around the run. Your search function, search, and your agent, rag_agent, are your own code:

import rius
from agents import Agent, Runner, set_tracing_disabled
from rius import SpanKind
 
rius.init(api_key="ri_...", service_name="docs-chatbot")
set_tracing_disabled(True)
 
rag_agent = Agent(
    name="rag_agent",
    instructions="Answer from the provided context only.",
    model="gpt-5-mini",
)
 
@rius.observe(name="answer-question", kind=SpanKind.AGENT)
def answer(question: str) -> str:
    with rius.start_as_current_span(
        kind=SpanKind.RETRIEVER, data_source_id="product-kb", top_k=5
    ) as obs:
        obs.set_input(question)
        docs = search(question)
        obs.set_retrieved_documents([{"id": d.id, "score": d.score} for d in docs])
    context = "\n\n".join(d.text for d in docs)
    result = Runner.run_sync(
        rag_agent, f"Context:\n{context}\n\nQuestion: {question}"
    )
    return str(result.final_output)
 
def handle_turn(conversation_id: str, user_id: str, question: str) -> str:
    with rius.session(conversation_id), rius.user(user_id):
        return answer(question)

4. If the agent calls retrieval as a function tool instead, put @function_tool outermost and @rius.observe(kind=SpanKind.TOOL) on the function body. Open the retrieval span inside that body.

5. Set the session and user on every request. Scope rius.session() with your conversation id so each turn of one chat groups together. Scope rius.user() with an opaque user id rather than an email. Both scopes wrap the root span, so every span in the answer carries them, including the model calls the integration records. Do not set a process-wide session in a server that handles many users, because their traces would merge into one session.

6. Run one question and open the trace in the console. You should see this shape:

answer-question          AGENT
├─ retrieval product-kb  RETRIEVER
└─ Response              LLM

What you get

• One trace per answer, with the root span showing the whole run and the retrieval span showing which source was searched, how many documents were asked for, and their ids and scores.
• A generation for each model call with model, tokens, cost and time to first token. Cost is a list-price estimate in US dollars, computed by Rius from the model and token counts, so you never send a cost value.
• Named tool spans that the Tool kind filter and the Tool Loop alert can see.
• Turns grouped by conversation through the session, and the Users page showing each person’s traces, failures and cost through the user id.
• A view of where an answer went wrong. A weak answer with good retrieval scores points at the model call. Poor scores point at the index.

When something else fits

• Handoffs between agents produce no span of their own, and every Responses API call is named Response. If you need agent boundaries visible, add an observe wrapper per agent entry.
• For a TypeScript chatbot, there is no OpenAI Agents SDK integration. Trace the model calls with manual generations, or see the Integrations matrix for the framework you use.

Related