How do I set the prompt cache token attribute on an OpenTelemetry span in Python?

Rius reads prompt cache tokens from two span attributes: gen_ai.usage.cache_read.input_tokens for cache reads and gen_ai.usage.cache_write.input_tokens for cache writes. In Python you set them with span.set_attributes from the OpenTelemetry API, next to the model name and the input and output token counts. Rius then prices the cache reads and writes at their own rates and computes the span’s cost.

How it works

The OpenTelemetry GenAI conventions define one key for each direction of prompt caching, both integers:

• gen_ai.usage.cache_read.input_tokens: prompt tokens served from the cache.
• gen_ai.usage.cache_write.input_tokens: prompt tokens written to the cache.

Both are subsets of the input count. Set gen_ai.usage.input_tokens to the full prompt size, cached tokens included, and the cache keys to the cached part of it. Reasoning tokens work the same way: gen_ai.usage.reasoning.output_tokens is a subset of the output count.

Rius computes cost on the server from gen_ai.request.model plus the gen_ai.usage.* counts: input, output, cache read and cache write, each at its own rate. A span needs the request model and at least one count, and any count you leave out is read as 0. A span with no model, or a model the price catalog does not know, gets no cost. Never send a cost attribute. Send the model name and the token counts.

If your instrumentation already uses another spelling, Rius maps it for you at ingest. The older gen_ai.usage.cache_creation.input_tokens becomes the cache write key, and Pydantic AI’s gen_ai.usage.details.cache_read_tokens and cache_write_tokens become the cache read and write keys. OpenInference’s llm.token_count.* cache details are mapped too. Native gen_ai.* keys always win over mapped ones. When you write the spans yourself, use the canonical keys directly.

Set it up

1. Register a tracer provider with the OTLP HTTP exporter and your API key, as shown on Vanilla OpenTelemetry SDKs. The OpenTelemetry API only records spans once a provider is registered. The Trace LLM calls with OpenTelemetry guide covers the whole path.

2. Open a span for the model call and set the request attributes before you call the model. Name the span the way the GenAI conventions ask: the operation, then the model.

3. After the call, set the usage attributes, including the two cache counts. input_tokens, output_tokens, cache_read_tokens and cache_write_tokens below are your own variables, filled from your provider’s response.

from opentelemetry import trace
 
tracer = trace.get_tracer("my-agent")
 
with tracer.start_as_current_span("chat gpt-4o") as span:
    span.set_attributes({
        "openinference.span.kind": "LLM",
        "gen_ai.operation.name": "chat",
        "gen_ai.request.model": "gpt-4o",
        "gen_ai.provider.name": "openai",
    })
    response = call_the_model()
    span.set_attributes({
        # Prompt tokens, cached ones included
        "gen_ai.usage.input_tokens": input_tokens,
        "gen_ai.usage.output_tokens": output_tokens,
        # Subsets of the input count
        "gen_ai.usage.cache_read.input_tokens": cache_read_tokens,
        "gen_ai.usage.cache_write.input_tokens": cache_write_tokens,
        "gen_ai.response.model": response.model,
    })

4. Set the cache attributes as integers. If your provider reports no cache writes, leave gen_ai.usage.cache_write.input_tokens off rather than inventing a value. Rius reads a missing count as 0.

5. Run the agent and open the trace in the console. Only the keys you set carry data, so check that the model name is spelled as the provider spells it.

What you get

• Cost per call that prices cache reads and writes at their own rates, not at the plain input rate. Costs are list-price estimates in US dollars from LiteLLM’s public model-price catalog. An unknown model gets an empty cost, not zero.
• A full generation span that appears in model analytics, because it carries the model, the tokens and the cost.
• Cache counts you can search and filter on under their canonical keys. A filter on gen_ai.usage.cache_read.input_tokens matches. A filter on a source spelling such as llm.token_count.prompt or gen_ai.usage.prompt_tokens matches nothing, because Rius moves those values onto the canonical keys and removes the source keys.
• The same analytics as spans from the Rius SDKs. The full list of keys, with the message shape for prompts and replies, is in the span attribute reference.

When something else fits

• If you call a supported client library or framework from Python, the Rius SDK’s integrations record the model and token counts for you, so you do not set these keys by hand. See the span attribute reference for what each key does.
• If your spans come from OpenInference or Pydantic AI, you can leave their usage keys as they are, because Rius maps them onto the canonical keys at ingest.

Related