How do I trace streaming responses from the OpenAI API?

Rius traces streamed OpenAI calls automatically once init() runs, and adds token counts, cost and, in Python, time to first token. Set include_usage on the stream so the span carries token counts, because without them there is no cost. In TypeScript, streamed calls have no time to first token from the integration, so you trace those calls as a manual generation when you need it.

How it works

The OpenAI integration instruments the client library for the whole process. Every call is traced without changes to your call sites, streamed or not. The span for each model call carries the model, the input and output messages, and the token usage. Rius computes cost from the model and tokens, so you never send a cost value.

Streaming adds two details:

• Token usage. OpenAI does not include usage in a stream by default. It sends usage with the final chunk only when you ask for it with stream_options. Without it, the streamed span has no token counts and so no cost.
• Time to first token. The Python integration records it on streamed calls with no manual code. The TypeScript integration does not, because the OpenInference OpenAI instrumentation for Node does not record it. TypeScript streams still carry token usage and cost when you set include_usage.

If you stream through LangChain or LiteLLM in Python over the openai integration, the time to first token lands on the nested OpenAI span. For other LangChain and LiteLLM streams, trace the call as a manual generation.

Time to first token matters for streaming because total duration mostly reflects how long the answer is. Rius uses it for per-model time to first token percentiles and inter-token latency in your analytics.

Set it up

1. Install the integration and initialize the SDK. The extra installs the instrumentation, not openai itself. The OpenAI integration page covers the TypeScript package and the pnpm and serverless cases.

pip install "glassflow-rius[openai]" openai

import rius
from openai import OpenAI
 
rius.init(api_key="ri_...", service_name="my-agent")
client = OpenAI()  # use the client exactly as before

2. Ask OpenAI to send usage with the final chunk. This is the one change your streaming call needs:

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    stream=True,
    stream_options={"include_usage": True},
)

3. In Python, stop here. The integration records time to first token for you.

4. In TypeScript, or anywhere you need time to first token and the integration does not record it, trace the call as a manual generation. Open the generation, call recordFirstToken() for every chunk, then set the output and end the span yourself. The call is idempotent, so only the first chunk records anything. Put end() in a finally block so the span closes even if the stream fails:

import { startGeneration } from '@glassflow-ai/rius'
 
const gen = startGeneration({ model: 'gpt-4o', provider: 'openai' })
gen.setInput(messages)
 
const chunks: string[] = []
try {
  const stream = await client.chat.completions.create({
    model: 'gpt-4o',
    messages,
    stream: true,
    stream_options: { include_usage: true },
  })
  for await (const chunk of stream) {
    gen.recordFirstToken() // idempotent; call it unconditionally per chunk
    const delta = chunk.choices[0]?.delta.content
    if (delta) chunks.push(delta)
    if (chunk.usage) {
      gen.setUsage({
        inputTokens: chunk.usage.prompt_tokens,
        outputTokens: chunk.usage.completion_tokens,
      })
    }
  }
  gen.setOutput(chunks.join(''))
} catch (error) {
  gen.recordException(error)
  throw error
} finally {
  gen.end()
}

When a manual generation replaces the auto-instrumented call, leave openai out of init({ instruments: [...] }), or both spans carry the call. The Python version uses rius.start_generation and gen.record_first_token() the same way. The tracing page shows both languages.

5. Run one streamed call and open the trace in the console. The span shows the resolved model, input and output token counts, and a cost. Wrap the call in observe to see it nested inside your agent’s run instead of as a standalone trace.

What you get

• A span for each streamed call with the model, messages, token counts and a cost. The cost is a list-price estimate in US dollars.
• Time to first token on streamed calls in Python, without manual code. In TypeScript you get it by tracing the call as a manual generation.
• Errors recorded too: a call OpenAI rejects, such as a 404 for an unknown model, a 429 or a timeout, is an error span with error.type set to the error class.
• Per-model time to first token percentiles and inter-token latency in your analytics, from the recorded first token.
• Prompt and response content under your privacy controls. capture_content=False (captureContent: false in TypeScript) and masking apply to streamed calls the same way as to any other.

When something else fits

If you stream from TypeScript and need time to first token, trace that call as a manual generation, as in step 4. For the full list of integrations and which streams carry time to first token, see Integrations.

Related