How do I monitor a browser automation agent in production?

Rius traces a browser automation agent as one trace per run, with a span for each model call and each tool step, so you can read a run from its first action to its last. Mark the run as an agent span and each browser action as a tool span, then watch the Traces and Agents views and the alerts. Cost, tokens and errors sit on the spans, and Rius alerts you when the same error recurs or the same tool repeats in a loop. This follows the general setup in Monitor AI agents in production.

How it works

A run of this kind of agent is readable when it has three layers of spans in one trace:

• The run. Put observe with kind=SpanKind.AGENT on the function that takes a task and drives the agent. This is the root of the waterfall. A bare observe records a CHAIN span instead.
• Each LLM call. If you call the model through a supported library such as Anthropic or OpenAI, with its instrumentation installed, each call becomes its own span with the model, token counts and cost. Rius computes cost from the model and tokens, as a list-price estimate in US dollars. You never send it.
• Each tool or retrieval step. Put observe with kind=SpanKind.TOOL on every function the agent can call, such as one that opens a page, clicks or reads content. A step that fetches context for the model can use kind=SpanKind.RETRIEVER.

The tool kind matters for alerts. The Tool Loop alert counts only spans of kind TOOL, so a tool traced with a bare observe is never counted.

Set it up

1. Install the SDK and start it once at process start. The Monitor AI agents in production guide covers the install and the extras for your LLM library in full.

import rius
 
client = rius.init(
    api_key="ri_...",
    service_name="my-agent",
)

2. Mark the run and each tool step. The names below are yours.

from rius import SpanKind
 
@rius.observe(kind=SpanKind.AGENT)
def run_task(task: str) -> str:
    ...
 
@rius.observe(kind=SpanKind.TOOL)
def click_element(selector: str) -> str:
    ...

3. Name the agent and tag its versions so you can compare releases. Set RIUS_AGENT_NAME if you want a name other than the service name, and set RIUS_MAIN_AGENT_VERSION for the agent definition (prompt, tools, policy):

export RIUS_MAIN_AGENT_VERSION=prompt-v12

4. Call client.shutdown() when the process stops, including from your own SIGTERM handler. It sends the final stopped heartbeat, so a deploy does not show up as a crash.

5. In the console, create a distribution channel (Slack, email or a webhook) and attach it to each alert from that alert’s Delivers to section. Without a channel, a firing is only a row on the Alerts page.

What you get

• Traces. Each run is a waterfall from the root span down to every model call and tool step, with tokens, cost and errors on the spans. Open a failed run and see which step went wrong.
• Agents. The Agents page finds your agent from its traces and heartbeats. Scan the Errors and Cost columns, then open the agent and check:
  • Drift for a rising failed rate, time to first token or cost per trace.
  • Exceptions to see whether errors share one cause.
  • Versions to compare two releases of the agent.
  • Context for peak usage against the model’s context window.
• Failed versus recovered. Error rate counts runs with any errored span, while Failed traces counts runs whose root span ended in error. A high error rate with few failed traces means the agent recovers, at a cost in time and money.
• Instances. running, ready, stale, gone and stopped show whether each process is alive. A process that hangs but keeps sending heartbeats still shows as running, so heartbeats show liveness, not progress.
• Alerts. Consistent Error Pattern fires when the same error recurs across runs. Tool Loop fires when the same tool is called with identical arguments three or more times in a row within one trace, which needs TOOL spans. Swallowed Error catches a run that reports success while a step inside it that is not a tool call failed. A failed browser action is returned to the model as its result, so it does not count. You can add custom alerts on cost, duration, error rate and tokens.

When something else fits

• Rius traces, prices and alerts on agent runs. It does not offer evals, datasets, prompt management or guardrails, so use other tooling for those.
• If your agent has no heartbeat, because it runs in a short-lived runtime with the heartbeat off or has no API key, the Agents page marks it no heartbeat and lists no instances. Traces, cost and alerts still work.

Related