How do I monitor a document processing pipeline in production?

Rius shows each document run as one trace, with a span for the run, each model call and each tool step, plus views and alerts to watch once it is live. Wrap the run and each step with observe, and the model calls are traced for you with tokens and cost. Then watch the Traces, Home and Agents views, and route the pre-defined alerts to Slack, email or a webhook. The general setup is in Monitor AI agents in production.

How it works

A run of a document processing pipeline becomes one trace, read from the top down:

• The run. The function that takes one document and returns its result is the root. Wrapping it with observe records it, and passing kind=SpanKind.AGENT marks it as the agent’s entry point. A bare observe records a CHAIN span instead.
• Each model call. With the SDK installed and the instrumentation for your LLM library in place, each call becomes a span with its model, token counts and cost. Rius computes the cost from the model and tokens, so you never send it. Costs are list-price estimates in US dollars.
• Each retrieval or tool step. Wrap a step that searches or fetches documents with observe and kind=SpanKind.RETRIEVER, with a data_source_id. Only a function the model calls as a tool gets kind=SpanKind.TOOL, and it then appears as a tool span in the same trace. In TypeScript the form is kind: SpanKind.TOOL.

The tool kind matters for alerting. The Tool Loop alert only works on TOOL spans, so a step that is not marked as a tool is invisible to it.

Set it up

1. Create an account and an API key at the console, then install the SDK and the extra for your LLM library. The full walkthrough is in Monitor AI agents in production.

pip install "glassflow-rius[openai]" openai

2. Initialize once and wrap the function that handles one document. Give the agent a name so it is easy to find on the Agents page. The function name and path argument here are yours.

import rius
 
rius.init(
    api_key="ri_...",
    service_name="document-pipeline",
    agent_name="document-pipeline",
)
 
@rius.observe
def process_document(path: str) -> str:
    ...

3. Add kind=SpanKind.AGENT to that entry point, kind=SpanKind.RETRIEVER to each search or fetch step, and kind=SpanKind.TOOL to each tool the model calls. An AGENT span is named invoke_agent followed by the agent name, and a TOOL span execute_tool followed by the function or tool_name.

4. Tag each release so you can compare versions later. Set these in the environment of the process:

RIUS_MAIN_AGENT_VERSION=prompt-v12      # the agent definition: prompt, tools, policy
RIUS_SERVICE_VERSION=2026.09.29-a41f3c  # the build or deploy

5. If the pipeline is a short script, call shutdown() before it exits so queued spans are exported. In a long-running worker, call it from your signal handler, so a deploy shows an instance as stopped instead of gone.

6. Create a distribution channel (Slack, email or webhook) and attach it to each alert from its Delivers to section. Without a channel, a firing is only a row on the Alerts page.

What you get

• Traces. Each run is a waterfall from the root span down to every model call and tool step, with tokens, cost and errors on the spans. Open a slow or failed document and see which step took the time or broke.
• Home. Volume, cost by model, tokens and error rate per period, so you see when document volume or spend changes.
• Agents. Compare Error rate with Failed traces: a high error rate with few failed traces means the pipeline recovers, for example with retries, at a cost in time and money. Exceptions groups errors by failure type, Peak usage in Context shows how close long documents come to the model’s context window, and Versions and Drift show whether the last release changed failed rate or cost per trace.
• Liveness. Instances shows whether each worker is running, ready, stopped or gone. A hung process that still sends heartbeats shows as running.
• Pre-defined alerts. Consistent Error Pattern fires when the same error recurs across runs. Swallowed Error fires when a run reports success while a step inside it that is not a tool call failed. Errored tool calls do not count, because the model saw the failure. Tool Loop fires when the same tool is called with identical arguments three or more times in a row within one trace.
• Custom alerts. Add rules of your own on cost, duration, error rate and other workspace metrics.

When something else fits

• Rius traces, prices and alerts on runs. It does not score the quality of extracted fields, offer evals or datasets, or detect a hung process, so keep those checks in your own tests.
• If your pipeline has no LLM calls and only runs fixed steps, a general monitoring tool for jobs may fit better.

Related