How do I monitor a helpdesk copilot for employees in production?

Rius monitors a helpdesk copilot for employees by tracing each run as one trace, with a span for every LLM call and every tool or lookup step. You then watch the runs in Traces and Agents, and let alerts tell you when something repeats. Cost per call is computed by Rius from the model and token counts, so you never send it. The setup is the same two lines of instrumentation as any other agent, plus a few choices about which steps you wrap.

How it works

A readable run of a helpdesk copilot is a tree of three kinds of span, all in one trace:

• The run. Wrap the copilot’s entry point, the function that takes an employee’s question and returns the answer, with observe and kind=SpanKind.AGENT. This is the root span. A run counts as failed when this root span ends in error.
• Each LLM call. With a global init(), every installed integration is turned on automatically, so each model call becomes a generation span with its model, token counts and cost.
• Each tool or retrieval step. Wrap a knowledge-base lookup or ticket search with observe and kind=SpanKind.RETRIEVER, with a data_source_id. Wrap a function the model calls as a tool, such as a status check, with kind=SpanKind.TOOL. A bare observe records a generic CHAIN span.

A TOOL span is named execute_tool followed by the function or tool_name, and the AGENT root invoke_agent followed by the agent name.

In the Traces view each run opens as a waterfall from the root span down to every generation and tool span, with token counts, cost and errors on the spans. Rius also finds your agent from its traces and heartbeats, so you do not register it.

Set it up

1. Install the SDK and initialize it once at startup. The full walkthrough, including the integration for your LLM library, is in Monitor AI agents in production.

   import rius
    
   client = rius.init(
       api_key="ri_...",
       service_name="my-agent",
   )

2. Mark the run and its steps. Put kind=SpanKind.AGENT on the entry point kind=SpanKind.RETRIEVER on each lookup, and kind=SpanKind.TOOL on each tool the model calls. Use kind: SpanKind.TOOL in TypeScript. Tool spans matter for alerting too, because the Tool Loop detector needs them.

3. Give the copilot a stable name and tag each release, so you can compare versions later. Set agent_name in init or RIUS_AGENT_NAME, and set the versions in the environment:

   RIUS_MAIN_AGENT_VERSION=prompt-v12      # the agent definition: prompt, tools, policy
   RIUS_SERVICE_VERSION=2026.09.29-a41f3c  # the build or deploy

4. Handle shutdown. Heartbeats are on by default. Call client.shutdown() from your SIGTERM handler, so a deploy ends an instance as stopped instead of gone. In TypeScript, a short script must also await shutdown() or its spans are never sent.

5. Create a distribution channel. A workspace admin adds a Slack, email or webhook channel and attaches it to each alert from that alert’s Delivers to section. Without a channel, a firing is only a row on the Alerts page.

6. Verify. Run the copilot, open Agents, and check that it has an instance in running or ready. When the process exits normally, it moves to stopped.

What you get

Once the copilot is live, these are the views and alerts worth checking:

• Traces. Open a bad answer as a waterfall. You see which retrieval step ran, what each model call cost and where an error occurred.
• Agents list. Scan the Errors and Cost columns. Rows with problems come first, and an agent that sends traces but no heartbeat is marked no heartbeat. See Agents for every tile.
• Error rate against failed traces. A high error rate with few failed traces means the copilot recovers, for example through retries, at a cost in time and money. A failed trace is a run whose root span ended in error.
• Exceptions. Errored spans grouped by failure type, so you can tell one cause from several.
• Versions and Drift. After a prompt or model change, compare two versions on failed rate, cost per trace, latency and tokens per trace. Rius warns when either side has fewer than 20 traces. Drift charts show time to first token, cost per trace and failed rate over time.
• Context. Peak usage shows how close the largest call came to the model’s context window.
• Instances. Three replicas show as three instances under one name. gone usually points to a crash, an out-of-memory kill or a deploy that sent SIGTERM.
• Pre-defined alerts. Consistent Error Pattern fires when the same error recurs across runs. Swallowed Error fires when a run reports success while a step inside it that is not a tool call failed. Errored tool calls do not count, because the model saw the failure. Tool Loop fires when the same tool is called with identical arguments three or more times in a row within one trace. You do not build these.
• Custom alerts. Add rules on workspace metrics for thresholds of your own, such as cost or error rate.

Costs are list-price estimates in US dollars, not billed amounts.

When something else fits

• Heartbeats show that a process is alive, not that it is making progress. A hung copilot that still sends heartbeats shows as running, so watch for runs that stay open.
• Rius traces, prices and alerts on agent runs. It does not score answer quality, run evals, or offer guardrails or moderation, so use a separate tool for those.

Related