# Reliability

The design promise of the Rius SDK: **tracing never blocks and never
breaks your application**. This page explains the machinery behind that and
the knobs you can tune.

## The export pipeline

Spans are queued in-process and exported in batches from a background thread
(OpenTelemetry's `BatchSpanProcessor` with an OTLP/HTTP exporter). Span
creation is an in-memory operation; the network never sits on your agent's
hot path.

Standard OpenTelemetry environment variables tune the batching:

| Variable | Default | Meaning |
|---|---|---|
| `OTEL_BSP_MAX_QUEUE_SIZE` | `2048` | Spans buffered in memory. When full, new spans are dropped with a warning. |
| `OTEL_BSP_SCHEDULE_DELAY` | `5000` (ms) | How often a batch is flushed. |
| `OTEL_BSP_MAX_EXPORT_BATCH_SIZE` | `512` | Spans per HTTP request. |
| `OTEL_BSP_EXPORT_TIMEOUT` | `30000` (ms) | Timeout per export attempt. |

For very chatty agents, raise the queue size before lowering the sample
rate; for latency-sensitive shutdowns, lower the schedule delay.

## Retries and failure behavior

Both SDKs retry a failed export only when the failure is likely to pass:

| Failure | What happens |
|---|---|
| Backend unreachable, `502`, `503`, `504` (Python also `408` and every other `5xx`) | Retried with backoff until the export timeout, then the batch is dropped |
| `401`, `402`, and any other error | Dropped, not retried |
| `429` | Python drops the batch. TypeScript retries after the `Retry-After` delay when it fits within the export timeout (10 seconds by default), and drops the batch otherwise |

Nothing that is dropped is buffered for later. Two platform answers need
action from you (see
[the full list](/rius/interoperability/attributes#where-to-send-spans)):

- **`402`: billing blocks ingest.** The trial has ended, the subscription is
canceled, or the workspace is paused because the organization holds more
workspaces than its plan allows. Traces sent in that period are lost even
after billing is fixed. Check **Settings → Plan & usage**.
- **`429`: the workspace is over its plan's ingest rate.** The platform
refuses the whole request, not only the spans past the rate, and asks for
a delay of up to 60 seconds, often longer than the TypeScript SDK waits. So
sustained traffic above the rate loses spans in both SDKs. Lower the volume
with [sampling](/rius/sdk/advanced#sampling) or move to a plan with a
higher rate.

If the backend stays unreachable, spans are dropped and errors are logged.
Your application continues normally; exporter exceptions never propagate
into application code.

## Flushing and shutdown

Pending spans are flushed automatically at interpreter exit. For short-lived
scripts and batch jobs, or before a hard kill, flush explicitly:

<Tabs items={['Python', 'TypeScript']} storageKey="sdk-language">
<Tabs.Tab>
```python
client = rius.init(...)
...
client.flush(timeout_millis=30_000)   # force an export
client.shutdown()                     # drain, stop the thread, release init()
```
</Tabs.Tab>
<Tabs.Tab>
The TypeScript SDK registers no process-exit hook, so flushing is not optional:
call it before your process ends.

```typescript
const client = init({ /* ... */ })
// ...
const delivered = await client.flush()   // force an export
if (!delivered) console.warn('the most recent span export failed')

await client.shutdown()                  // drain, tear down, release init()
```
</Tabs.Tab>
</Tabs>

Since 0.9.0, `flush()` reports delivery, not just queue drain: it returns
`False` on timeout or when the most recent export was rejected (for example
a 401 on a bad API key), so `True` means your spans actually left. Earlier
releases returned `True` whenever the queue drained, even while every batch
was failing.

`shutdown()` also releases the global `init()` slot, which is what allows a
later `init()` to reconfigure the SDK.

The TypeScript pipeline is the same `BatchSpanProcessor` driven by a background
timer rather than a thread, tuned by the same `OTEL_BSP_*` variables, and it
retries transient failures as in the table above. Two more differences matter
in practice:
it installs no exit hook, so nothing is flushed for you, and only the first
export failure in a process is logged, naming `RIUS_API_KEY` and `RIUS_ENDPOINT`
as the likely cause. `flush()` resolving `false` is how you detect later
failures.

## Short-lived processes

Export happens on a background thread, so what you need to do depends on how
your process ends:

| Where your code runs | What to do | Why |
|---|---|---|
| Long-running server (FastAPI, worker, ...) | Python: nothing. TypeScript: `await client.shutdown()` on graceful shutdown, for example in a `SIGTERM` handler | Batches export continuously. In Python the exit hook drains the rest on clean shutdown; TypeScript installs no exit hook. |
| CLI or batch job | `client.flush()` before exiting (in Python you can rely on the exit hook instead) | Short processes can finish before the first scheduled batch export fires. |
| Serverless (Lambda, Cloud Functions, ...) | `client.flush()` at the end of **every** invocation | The runtime freezes the process between invocations; a frozen background thread exports nothing, and the exit hook may never run. |
| Fork-based workers (gunicorn, multiprocessing) | Nothing at fork time; `flush()` only if workers are killed hard | The exporter re-creates its background thread in each forked child automatically; only an unclean worker shutdown can lose the last batch. |

Never call `flush()` inside request handlers or other hot paths: it blocks
until the export completes, which is exactly the stall the background
exporter exists to avoid. Flush at process or invocation boundaries only.

## Disabled mode

`disabled=True` (or `RIUS_DISABLED=1`) attaches no exporter at all:
spans are created and dropped in-process. Instrumented code runs unchanged,
which makes it the right kill switch for tests and incident response.

## Heartbeat delivery

With the [heartbeat enabled](/rius/sdk/installation#configuration-reference)
(the default in both SDKs), the SDK pings the platform for the process's
whole lifetime, completely independent of the trace pipeline above. Python
uses a daemon thread, TypeScript an unreferenced timer, and neither keeps
your process alive. The delivery rules are deliberately the opposite of the
exporter's:

- **Never queued, never retried.** Liveness is only true fresh; delivering
a heartbeat late would be misinformation. A failed ping is dropped, the
next interval tries again, and delivery problems warn once and then stay
quiet. Like everything in the SDK, failures never raise into your code.
- **The first ping is immediate**, so the agent appears in the
[Agents view](/rius/platform/agents) at `init()` rather than one interval
later.
- **A clean stop means calling `shutdown()`.** That sends the final
`stopped` ping, which is how the platform tells a clean exit from a crash.
Both SDKs also send it when the process ends naturally, and neither
installs signal handlers, because a library must not own process signals.
An unhandled `SIGTERM` or a `SIGKILL` therefore sends no `stopped` ping
and the instance reports `gone`. That is the design: `gone` is the crash
signal. In TypeScript the same applies to any abrupt exit, including
`process.exit()`, so call `client.shutdown()` if you want the distinction
to show up.
- **Each process has its own identity.** In Python a forked child re-arms
the heartbeat with a new instance ID, so a gunicorn app shows one instance
per worker. In TypeScript each process that calls `init()` gets its own
instance ID for the same reason: one identity never speaks for two
processes.
- **Serverless freezes the process.** Between invocations the runtime
suspends it, pings stop, and the instance flaps between alive and `stale`
or `gone`. Heartbeats are a long-lived-process signal and they are on by
default, so set `heartbeat=False` in Python or `heartbeat: false` in
TypeScript (or `RIUS_HEARTBEAT=false`) in Lambda-style runtimes.

The payload carries no span content: instance ID, agent name, send time,
SDK language and version, the IDs of currently open root traces (capped at
32) plus their true count, the `stopped` flag, and, when another
OpenTelemetry SDK holds the global tracer provider, a cumulative
`foreign_parent_spans` count. `capture_content` and `mask`
have nothing to strip here, but note that trace IDs do leave the process
in heartbeats, not only in spans.

## Partial spans

A span leaves the process when it ends, so a run that is still going is
invisible and a run that crashed exports nothing at all. With
[`partial_spans`](/rius/sdk/installation#configuration-reference) enabled
(`partialSpans` in TypeScript, off by default in both), every sampled span
also exports a snapshot at **start**. The platform replaces that snapshot
with the real span when the span ends, so a snapshot that is never replaced
is the durable record of what a crashed agent was doing.

- **Snapshots carry no content, ever.** They are filtered to identity
attributes: the span kind and operation, the provider, the tool name, tool
call id and tool type, the agent name, id, and version, the data source and
`top_k` of a retrieval, the MCP method and protocol version, the session
and user ids, the requested output type, and the request settings under
`gen_ai.request.*` and `rius.request.*`, such as the model. This is an
allowlist rather than a list of things to remove, so content set by
third-party instrumentation cannot ride along either. A snapshot is built once at span
start and never rebuilt, so content added later in the span's life cannot
reach it.
- **They share the export pipeline.** Snapshots go through the same batching,
retries and masking as finished spans, so nothing about the delivery rules
above changes.
- **`partial_spans_delay` keeps the volume sane.** Most agent spans live
milliseconds, and a snapshot for one of those is superseded almost
instantly. With a delay set (`partialSpansDelay` in TypeScript, up to 60
seconds), a snapshot waits and is only sent if the span is still open, so a
span that finishes first costs nothing on the wire. Anything worth watching
live stays open longer than the delay.
- **A snapshot is a zero-duration span.** It carries the same trace and span
IDs, parent, name and start time as the final span, with its end time equal
to its start time, which is how the platform matches the two. It is marked
with the `rius.span.pending` attribute (`glassflow.span.pending` before
1.0.0; the platform reads both).

## Next steps

- [Troubleshooting](/rius/sdk/troubleshooting): symptom-first diagnosis when
traces go missing.
- [Advanced features](/rius/sdk/advanced): sampling, when the volume rather
than the pipeline is the problem.
