How do I monitor a voice assistant in production?

Rius monitors a voice assistant in production by tracing each run, each LLM call and each tool or retrieval step, then alerting on recurring errors and tool loops. Wrap the assistant’s entry point as an agent span and its tools as tool spans. Rius records the model, tokens and cost on each LLM call for you. Once it is live, the Traces, Agents and Alerts views show whether runs fail, slow down or cost more.

How it works

A run of a voice assistant becomes one trace, a waterfall from the root span down to each step. Four kinds of span make it readable:

• The run. Wrap the assistant’s entry point with observe and kind=SpanKind.AGENT. Rius records an agent span named after the agent. A failed trace is a run whose root span ended in error.
• Each LLM call. With the instrumentation for your LLM library installed, every model call becomes a generation span with its model, token counts and cost. Rius computes cost from the model and tokens as a list-price estimate in US dollars, so you never send it.
• Each tool step. Wrap each tool function with observe and kind=SpanKind.TOOL. Rius records a tool span named after the tool. A bare observe makes a CHAIN span, which the Tool kind filter and the Tool Loop alert never see.
• Each retrieval step. Use kind=SpanKind.RETRIEVER on a lookup function to get a retrieval span.

If you build the assistant on a supported framework, its integration emits its own tool and agent spans. The Monitor AI agents in production guide covers the full instrumentation walkthrough.

Set it up

1. Install the SDK with the extra for the LLM library you use, then call init. The extra installs the instrumentation, not the library, so name both. Replace openai with your library’s extra and package if it differs.

pip install "glassflow-rius[openai]" openai

import rius
 
client = rius.init(
    api_key="ri_...",
    service_name="my-agent",
)

2. Add kind=SpanKind.AGENT to the entry point that handles one turn or one call, kind=SpanKind.TOOL to each tool function, and kind=SpanKind.RETRIEVER to each retrieval function. In TypeScript the form is kind: SpanKind.TOOL.
3. Tag each release so you can compare versions later:

export RIUS_MAIN_AGENT_VERSION=prompt-v12

4. Heartbeats are on by default and need your API key. They show whether each process is alive on the Agents page. Call client.shutdown() from your SIGTERM handler. A process stopped by a deploy then ends as stopped instead of gone, which you would otherwise read as a crash.
5. Create a distribution channel (Slack, email or webhook) as a workspace admin. Attach it to each alert from that alert’s Delivers to section. Without a channel, a firing is only a row on the Alerts page.

What you get

• Readable runs. Open any trace to see the agent span, every model call with tokens and cost, and every tool and retrieval step with its errors.
• Per-agent health. The Agents page shows failed traces, error rate, cost and time to first token (p50). The time to first token counts streaming LLM calls only, and the tile shows how many calls were measured. Drift charts show whether time to first token, cost per trace or failed rate is rising.
• Failed versus recovered. Compare Error rate with Failed traces. A high error rate with few failed traces means the assistant recovers, for example with retries, at a cost in time and money.
• Release checks. With a version set, Compare puts two versions side by side on failed rate, cost per trace, latency and tokens. It warns when either side has fewer than 20 traces.
• Context pressure. Peak usage shows how close the largest call came to the model’s context window.
• Crash versus shutdown. Instances show stopped for a clean exit and gone for a process killed without a final heartbeat.
• Alerts without setup. Consistent Error Pattern fires when the same error recurs across runs. Swallowed Error fires when a run reports success while a step inside it failed. Tool Loop fires when the same tool is called with identical arguments three or more times in a row within one trace, and needs tool spans. Add custom alerts on cost, duration or error rate for thresholds of your own.

When something else fits

Heartbeats show that a process is alive, not that it is making progress. A hung process that still sends heartbeats shows as running, so pair the Agents page with alerts on duration or errors. Rius traces, prices and alerts on runs. It does not offer scoring or evals of answers, or guardrails and moderation, so use a separate tool if you need those.

Related