Skip to Content

My CrewAI agent got worse after a release. Why?

Rius shows what changed after a CrewAI release by comparing agent versions on failed rate, cost per trace, latency and tokens, then lets you open the traces behind the difference. Tag each release with a version, open the agent on the Agents page, and compare the new version with the old one in Versions and Drift. Then read the model calls and tool spans in the slow, costly or failed traces to see which step moved. This is the same loop described in Monitor AI agents in production.

How it works

Rius cannot say why a release regressed on its own, but it narrows the search to the runs and steps that changed.

  • Compare versions. On the agent’s page, Versions puts a base version next to another and shows traces, failed rate, cost per trace, latency, time to first token (p50) and tokens per trace, with the change. It warns when either side has fewer than 20 traces, because the difference can then be noise. See Agents.
  • Find when it started. Drift charts time to first token, cost per trace and failed rate over time, so you can see whether the change lines up with the deploy.
  • Group the failures. Exceptions groups errored spans by failure type. A single group points at one cause, several groups point at a broader problem. Comparing Error rate with Failed traces tells you whether the agent is recovering with retries (slower and costlier) or actually breaking.
  • Read the traces. The CrewAI integration records each model call as a generation with model, tokens, cost and time to first token. Your observe wrappers give the run its root span and each tool its own span. A run that calls a tool looks like an agent root, a model call that decides to use the tool, the tool span, then a model call that answers.

In a trace, the waterfall shows where time and failures sit. For a trace with several model calls, the context growth strip plots input tokens per call, which is where a prompt that grows after a release shows first. Since previous llm call on a span shows what each step added to the prompt, so a rewritten system prompt or a dropped turn stands out. Compare runs sets one run against another run of the same agent and shows which steps changed, were added or dropped, and where cost or latency moved. See Exploring traces.

Crews and tasks produce no spans of their own, and generation spans share the generic name ChatCompletion. You cannot tell from the trace which task a call belonged to unless you wrap per-task entrypoints with observe. Details are on the CrewAI integration page.

Set it up

  1. Install the OpenAI instrumentation next to CrewAI, which captures its model calls, and start Rius. The Integrations page covers the full setup.

    pip install "glassflow-rius[openai]" crewai
  2. Tag every release so versions can be compared. Use RIUS_MAIN_AGENT_VERSION for the agent definition (prompt, tools, policy) and RIUS_SERVICE_VERSION for the build.

    RIUS_MAIN_AGENT_VERSION=prompt-v12 # the agent definition: prompt, tools, policy RIUS_SERVICE_VERSION=2026.09.29-a41f3c # the build or deploy
  3. Wrap the entrypoint as an agent and each tool body as a tool, with @tool outermost. The tool kind matters: a bare observe makes a chain span that the Tool kind filter and the Tool Loop alert never see.

    import rius from crewai import Agent, Crew, Task from crewai.tools import tool from rius import SpanKind rius.init(api_key="ri_...", service_name="my-agent") def lookup(email: str) -> str: return "active" # replace with your account lookup @tool("account_status") @rius.observe(kind=SpanKind.TOOL) def account_status(email: str) -> str: """Fetch a customer's account standing by email.""" return lookup(email) support_agent = Agent( role="Support specialist", goal="Answer support tickets accurately and concisely.", backstory="You resolve product and account questions.", tools=[account_status], llm="gpt-5-mini", ) @rius.observe(name="handle-ticket", kind=SpanKind.AGENT) def handle_ticket(ticket: str) -> str: task = Task( description=f"Resolve this support ticket: {ticket}", expected_output="A short, direct answer.", agent=support_agent, ) return str(Crew(agents=[support_agent], tasks=[task]).kickoff())
  4. After the release, open the agent on Agents, set the time window, and use Compare in Versions with the previous version as the base. Wait until each side has at least 20 traces before trusting the numbers.

  5. Click View all in Traces from Recent traces or Errored traces, then sort the Spans tab by Cost or Duration and filter by kind with All kinds to find the step that changed.

  6. Set CREWAI_TRACING_ENABLED=false where no terminal is attached, because CrewAI asks about its own tracing on first run.

What you get

  • A side-by-side answer to “did the release help or hurt” on failed rate, cost per trace, latency, time to first token and tokens per trace.
  • A chart of when the change began, so you can match it to a deploy rather than guess.
  • Failures grouped by cause, and a clear split between runs that recovered and runs that failed.
  • Per-call model, token and cost detail. Costs are list-price estimates in US dollars, not billed amounts.
  • Retried tool calls collapsed into one row such as fetch_article · 3 attempts · recovered, so a new retry loop reads as one event.
  • Next time, an alert before users notice. The pre-defined alerts are Consistent Error Pattern, Swallowed Error, Tool Loop and Context rot. Tool Loop counts only spans of kind TOOL, which is why step 3 sets that kind. For cost, create a custom alert filtered by Model, or by Span kind LLM, because tokens and cost sit on model-call spans. Filter by Service or Agent to measure whole runs.

When something else fits

  • If the agent’s own work is hard to follow because tasks have no spans, wrap per-task entrypoints with observe before you compare versions.
  • If you installed LiteLLM alongside CrewAI, use the LiteLLM integration for its model calls instead of the OpenAI one.
  • If the new version has fewer than 20 traces, wait for more runs, or read individual traces with Compare runs instead of relying on the version totals.

Still stuck? Get help

Last updated on