How do I get alerted when my agent crashes and stops sending traces?

Rius alerts you when an agent goes silent with a custom alert on Traces count below 1, scoped to that agent’s service. The rule fires when no traces arrive in its window, so a crashed process triggers it even though the crash itself produces no error. When it fires, the Agents page shows whether the process exited cleanly or died, so you know where to look.

How it works

The pre-defined alerts cover recurring errors, swallowed errors, tool loops and context rot. They watch for things that happen in traces, so they do not cover a process that has stopped sending anything. The alert that fits is a custom alert that works as a dead-man switch.

• The rule: a Traces count < 1 rule over a generous window fires when ingestion stops. A custom alert is one or more rules that all share one rolling window, and the window runs from 5 minutes to 24 hours.
• The scope: add a Service filter, for example Service = checkout-agent. Filters apply before aggregation, so the rule watches only that agent’s spans. Without a filter, the rule fires only when the whole workspace goes quiet.
• The window: pick one longer than the longest normal gap between runs of your agent. A bursty agent with a 5 minute window fires between bursts.
• The cadence: Rius evaluates a rule every minute while it has no open firing, and every five minutes while one is open, so timing is approximate to that.
• The lifecycle: a firing stays open while the condition matches. It resolves on its own once the condition has not matched for two hours. If the agent goes quiet again later, that is a new firing.

The alert tells you that traces stopped. The Agents page tells you why. Each running process reports as an instance through heartbeats. An instance that stops sending heartbeats for 60 seconds without a final heartbeat is gone, which usually points to a crash, an out-of-memory kill or a deploy that sent SIGTERM. An instance that sent its final heartbeat before exiting is stopped, a normal shutdown.

Set it up

1. Send heartbeats so the Agents page can tell gone from stopped. Heartbeats are on by default in both SDKs and need an API key. Call shutdown() on exit, and from your own SIGTERM handler, because the SDK installs no signal handlers:

import rius
 
client = rius.init(
    api_key="ri_...",
    service_name="checkout-agent",
    agent_name="checkout-agent",
    heartbeat_interval=15,
)
 
# ... run your agent ...
 
client.shutdown()  # sends the final `stopped` heartbeat

Keep heartbeat_interval at 15 seconds or below. Rius marks an instance stale at 30 seconds and gone at 60 seconds, and a longer interval makes a live instance show as stale between heartbeats.

2. Open Alerts and create a custom alert. You need admin rights, because creating alerts is an admin action. Add the filter, the rule and the window in the builder:

FILTERS   Service = checkout-agent
RULES     count(Traces) < 1
WINDOW    60 minutes

3. Attach a distribution channel (webhook, email or Slack) so the firing reaches you.

4. Set Renotify to every N minutes, from 5 to 1440. By default Rius sends one notification per firing. With renotify on, it repeats the notification until someone acknowledges the firing or it resolves.

5. Check the builder’s plain-language preview. It states exactly when the alert fires, and becomes the alert’s description on its details page.

6. Verify the heartbeat side. Start your agent, open Agents and confirm it has an instance in running or ready. Let the process exit and the instance moves to stopped. If it goes to stale and then gone, the process was killed before it could send the final heartbeat.

What you get

• A firing, with a notification on each channel you attached, when the agent sends no traces for the whole window.
• A firing page showing when it opened, when the condition was last seen, the observed value against your threshold, and one delivery row per channel and attempt.
• Acknowledgment for your team. Acknowledging records who saw the firing and stops further notifications for it. The firing still resolves on its own once traces have been arriving again for two hours.
• A place to investigate. On the agent’s page, the Instances section shows gone or stopped for each process, Lifecycle events lists when each instance started, finished, went quiet or came back, and Recent traces shows the last runs. A trace marked frozen? is still running on an instance that has gone stale.
• Evidence of the cause. A gone instance stays visible for a week after its last heartbeat, long enough to investigate.

When something else fits

• An agent that sends no heartbeat has no instances, and so does one started without an API key or with heartbeats off. The Agents page then cannot show gone or stopped, but the Traces count rule still works.
• A hung process that keeps sending heartbeats shows as running, because heartbeats show liveness, not progress. The Traces count rule covers that case only once the traces stop.
• If your agent sends traces but fails, use the pre-defined Consistent Error Pattern or Swallowed Error alerts, or a custom rule on Error rate.

Related