My LangChain agent got worse after a release. Why?
Rius shows what changed after a release by comparing agent versions and by recording each LangChain step as its own span, so you can see whether failures, cost or prompt size grew. Tag each release with a version, open the agent on the Agents page and compare the new version with the old one. Then open a failed trace to find the step that changed. Rius also has alerts that catch the same symptom next time.
How it works
The LangChain integration hooks LangChain’s callbacks, so chains, agents and LangGraph nodes are traced. Each step becomes a span of its own kind: chains as CHAIN, tool calls as TOOL, retrievers as RETRIEVER, and model calls as LLM spans with model, token usage and messages. Rius computes cost from the model and token counts. The nesting follows your graph, so a trace reads like your execution. See LangChain for what lands in the trace.
That structure tells you where “worse” comes from:
- Failed traces rose. A trace is failed when its root span ended in error. A trace is recovered when a child step errored but the run finished clean, for example after a retry or fallback. A rising error rate with steady failed traces means more retries, which costs time and money without breaking the result.
- Cost per trace rose. Look at the LLM spans. A new prompt, a different model or a longer history shows up as more input tokens per call.
- Tool calls repeat. Back-to-back calls of the same tool with the same input collapse into one row such as
fetch_article · 3 attempts · recovered. - Context grows. A trace with several model calls plots input tokens per call, so a runaway history stands out.
Costs are list-price estimates in US dollars. A call that reports no token usage shows a blank cost, not zero.
Set it up
- Tag each release so Rius can tell versions apart. Set
RIUS_MAIN_AGENT_VERSIONfor the agent definition (prompt, tools, policy) andRIUS_SERVICE_VERSIONfor the build. If the LangChain integration is not installed yet, the first line adds it. The full setup is on LangChain and Integrations.
pip install "glassflow-rius[langchain]" langchain
export RIUS_MAIN_AGENT_VERSION=prompt-v12
export RIUS_SERVICE_VERSION=2026.09.29-a41f3c- Open Agents, pick your agent and set the time window to cover both releases. Check Drift for a rising failed rate, cost per trace or time to first token. Check Exceptions to see whether the errors share one cause or have several.
- In Versions, use Compare to put the old version next to the new one. It shows traces, failed rate, cost per trace, latency, time to first token and tokens per trace, with the change. It warns when either side has fewer than 20 traces, because the difference can then be noise. Details are on Agents.
- Open Recent traces or Errored traces and click a failed trace. On the trace page, Exploring traces covers the views you need:
- The waterfall highlights the path from the root to the failed span.
- Compare runs sets the open trace against another run of the same agent and shows which steps changed, were added or were dropped.
- On an LLM span, Since previous llm call in the Input / Output tab marks messages as added, removed or changed. A dropped turn or a rewritten system prompt stands out.
- For the next release, set up alerts. Rius ships four pre-defined alerts: Consistent Error Pattern, Swallowed Error, Tool Loop and Context rot. Tool Loop counts only TOOL spans, which the LangChain integration records for tool calls. For cost or token growth, add a custom alert. To alert on model spend, filter on Model, or set Span kind to LLM. A filter that selects only tool, chain or agent spans measures no tokens and no cost.
For the wider routine around this, see Monitor AI agents in production.
What you get
- A side-by-side comparison of the old and new version on failed rate, cost per trace, latency and tokens, so you know whether the release hurt and by how much.
- Drift charts that show the day the change started.
- A nested trace per run, so you can go from “errors went up” to the exact chain, tool or model call that changed.
- Prompt-level diffs between model calls inside one run, when the SDK captures content.
- Alerts that notify you through webhook, email or Slack when the same pattern returns.
One caution when cost looks higher after a release. Text-completion models (BaseLLM subclasses, such as langchain_openai.OpenAI) are still counted twice when you also run a client instrumentation. If your release switched to one, part of the jump may be double counting. Keep one layer for those calls.
When something else fits
- If the agent has no version tag, set one first. Until then, compare by time window on Drift instead.
- If a process hangs but keeps sending heartbeats, it shows as running. Look for traces that stay at running… for too long instead of relying on instance status.
- If the symptom is answer quality with no errors, Rius shows what changed in the prompts, tools and cost. It does not score answers.