My AI agent crashes and stops sending traces. How do I find the cause?

Rius shows a crashed agent on the Agents page as an instance that ends as gone, and the spans of the run that was in flight point to the cause. Open the agent, check its Instances, then open the last traces it sent. A trace still showing running… or an error span with its traceback shows where the run died.

With heartbeats and partial spans on, you see both the moment the process vanished and the last step it reported.

How it works

Rius tells a crash from a clean shutdown by the last heartbeat of each process. Open Agents, then the agent, then read Instances:

• stopped: the process exited and sent its final heartbeat. This is a normal shutdown.
• stale: no heartbeat for 30 seconds.
• gone: no heartbeat for 60 seconds and no final heartbeat. This usually points to a crash, an out-of-memory kill, or a deploy that sent SIGTERM. It also happens when the process exits with process.exit() or os._exit().

A gone instance stays visible for a week after its last heartbeat, so you can investigate after the fact. Lifecycle events on the same page lists when each instance started, finished, went quiet or came back, so you can match the time it vanished against a deploy or a memory spike.

Then look at what the process was doing. Under Recent traces, a row marked frozen? is still running, but the instance running it has gone stale. That is the run that was in flight when the process died.

In the trace list, a run reads running… when partial spans are enabled. Filter by the incomplete status to find runs that started and never ended. A trace still showing running… long after the work should have finished is a crash, a hang, or a process killed before it could report. Without partial spans, a span reaches Rius once, when it ends, so a crashed run never arrives at all.

The spans that did arrive carry the clues:

• Exceptions on the agent page groups errored spans by failure type, such as llm request failed, so you can see whether one cause or several sit behind the errors.
• In the waterfall, failed spans are highlighted from the root to the error, and the detail panel of an error span leads with the exception and traceback.
• Context growth, the strip above the tree, plots input tokens per call. A prompt that keeps growing until the last call points to a context or memory problem. Peak usage in Context on the agent page near 100% points the same way.
• Retry chains collapse repeated calls into one row, such as fetch_article · 3 attempts · recovered, so a retry loop before the crash reads as one event.

Set it up

1. Turn on partial spans, so a run still going is visible and a crashed run leaves a record. The snapshot has no content.

export RIUS_PARTIAL_SPANS=true

2. Keep heartbeats on. They are on by default in both SDKs, but they need an API key or an Authorization header. Without credentials the agent shows no heartbeat and has no Instances. Call shutdown() so a clean exit ends as stopped:

import rius
 
client = rius.init(
    api_key="ri_...",
    service_name="checkout-agent",
    agent_name="checkout-agent",
    heartbeat_interval=15,
)
 
# ... run your agent ...
 
client.shutdown()  # sends the final `stopped` heartbeat

3. Call shutdown() from your own SIGTERM handler. The SDK installs no signal handlers, so a deploy that sends SIGTERM otherwise ends as gone, and you cannot tell it from a crash. In TypeScript, call await client.shutdown() before process.exit().
4. Keep heartbeat_interval at 15 seconds or below. Rius marks an instance stale at 30 seconds and gone at 60, so a longer interval makes a live process look stale between heartbeats.
5. Set RIUS_MAIN_AGENT_VERSION so you can see whether crashes began with a release. The agent page compares versions and plots drift in failed rate.

For the full picture of watching agents in production, see Monitor AI agents in production.

What you get

• A clear split between a clean stop and a crash: stopped against gone, with the time the instance went quiet.
• The run that was in flight, shown as running… or frozen?, instead of a gap in your data.
• The failing step, with the exception and traceback on the error span, plus grouped failure types under Exceptions.
• Signals for resource problems: input tokens growing call by call, and peak context usage close to the window.
• A timeline of instance starts, finishes and silences under Lifecycle events.
• Version comparison, so you can tell whether a release introduced the crashes.

When something else fits

Heartbeats show whether a process is alive, not whether it is stuck, so a hung process that still sends heartbeats shows as running. For a hang, look for traces that stay running… for too long. To find why a process was killed, such as an out-of-memory kill, check your platform’s own process and container logs, because Rius shows when the process vanished and what it last reported.

Related