How do I trace streaming responses from LiteLLM?

Rius traces a streamed LiteLLM call with the litellm integration, and records time to first token on the nested OpenAI span or from a manual generation. Each routed completion becomes a span with the model, token usage and messages, and Rius computes cost from the model and tokens. Time to first token is recorded automatically only when LiteLLM streams through the openai integration. For other LiteLLM streams, you trace the call as a manual generation and call record_first_token() from your loop.

How it works

• The span for the call. With the litellm extra installed, init() turns the integration on. Each routed completion becomes a span for the model call that carries the model, token usage and messages. Cost is computed by Rius, never sent by you.
• Time to first token. For a stream, total duration mostly reflects the length of the answer. Time to first token is the latency a user feels. When LiteLLM calls OpenAI or an OpenAI-compatible provider through the openai client and the openai extra is installed, that client call appears as a nested OpenAI span under the LiteLLM span, and the nested span gets time to first token automatically.
• Streams without it. Other LiteLLM streams carry no time to first token from auto-instrumentation. A stream that reaches a provider by another path therefore needs the manual generation below.
• Token usage on the nested span. That nested span comes from the OpenAI integration, which records token counts on a stream only when the call sets stream_options with include_usage true. Without it, that span has no token counts and so no cost.
• Counting. The call is counted once in tokens and cost, and the LiteLLM span still shows the run’s total. A few setups, such as streamed calls on some instrumentation versions, are still counted twice.
• When the span ends. In a manual generation you own the lifetime. The span ends when you call end(), so set the output after the loop and then end it.

Set it up

1. Install the integration and call init(). The LiteLLM page covers this in full, and the extra installs the instrumentation, not litellm itself:

   pip install "glassflow-rius[litellm]" litellm

2. Add the openai extra too if you route to OpenAI, so the nested OpenAI span exists and can carry time to first token. For OpenAI-compatible streams, set stream_options with include_usage true if you want token counts on that span.

3. Run a streamed completion and open the trace in the console. If the nested OpenAI span shows time to first token, you are done.

4. If the stream has no time to first token, trace the call as a manual generation. Turn the litellm and openai integrations off for this process, with instruments=[] or a list without them, or the call is counted twice. Calls you do not wrap are then untraced. record_first_token() is idempotent, so call it for every chunk without a guard. It records the moment the first chunk arrives and marks the call as streamed:

   gen = rius.start_generation(model="gpt-4o", provider="openai")
   gen.set_input(messages)
    
   chunks, usage = [], None
   try:
       stream = litellm.completion(
           model="gpt-4o", messages=messages, stream=True,
           stream_options={"include_usage": True},
       )
       for chunk in stream:
           gen.record_first_token()   # idempotent; call it unconditionally per chunk
           if chunk.choices and chunk.choices[0].delta.content:
               chunks.append(chunk.choices[0].delta.content)
           if getattr(chunk, "usage", None):
               usage = chunk.usage   # the final chunk carries it
       gen.set_output("".join(chunks))
       if usage:
           gen.set_usage(input_tokens=usage.prompt_tokens, output_tokens=usage.completion_tokens)
   finally:
       gen.end()

5. Pass the model and provider you actually call. For failures you catch, start_generation() also gives you record_exception(). The tracing guide covers generations and streaming in more detail.

What you get

• A span for each streamed LiteLLM call with the model, messages and, where the provider reports them, token usage, plus a cost estimate from Rius.
• Time to first token for streams over the openai integration, and for any stream you trace as a manual generation. Rius derives per-model time to first token percentiles and inter-token latency from it.
• One model name per provider. LiteLLM’s routing prefix is dropped when it names the provider, so openai/gpt-4o-mini lands as gpt-4o-mini with the provider openai.
• The same tracing when you switch providers, because one extra covers every provider behind the router.

The integrations overview lists the other known limits for streams.

When something else fits

• The LiteLLM integration is Python only. From TypeScript, calls to a LiteLLM proxy over its OpenAI-compatible endpoint are traced by the OpenAI integration, and the model recorded is the one you ask the proxy for.
• A LiteLLM proxy server is a separate process. Instrument the proxy itself, or send its OpenTelemetry traces through the interoperability path.
• When you need time to first token on a stream that auto-instrumentation misses, a manual generation is the better fit.

Related