The recurring failure mode is frustratingly common: an agent is updated, standard evals stay green, but performance noticeably degrades a week later. Two real-world examples catalyzed the development of this tool:

- LangGraph PR #4486: Added node-level result caching, which quietly masked the repeated-sampling variance a regression test depended on, something a plain threshold check has no way to catch.

- CrewAI PR #6134: Fixed a bug where file tools leaked absolute filesystem paths in responses. A standard quality scorer never flags this because it only checks if the answer is correct, not what is hidden in the payload.

These examples are part of a larger validation pass encompassing 29 PRs and 239 rows across LangGraph, CrewAI, and the OpenAI Agents SDK. The full data breakdown is available in the pr-analysis.md documentation on Github.

Instead of grading a single response, agent-eval statistically compares two whole batches of runs. By running versions A and B 50 times each, the tool executes:
- A Mann-Whitney U test (requiring no assumption that scores are normally distributed).
- A bootstrap confidence interval to confirm if the shift is real.
- Cohen's d to determine if the shift is large enough to actually matter.

A p-value under 0.05 by itself is not enough to call it a regression. The effect size is required for definitive proof.

As part of statistical guardrails, two limits are utilized to ensure data integrity.
- Under 30 runs per version: Triggers a warning due to a drop in statistical power.
- Under 10 runs: Returns INSUFFICIENT_DATA rather than guessing at an answer.

Repo: https://github.com/RudrenduPaul/agent-eval
MCP Servers:
https://mcpservers.org/servers/rudrendupaul/agent-eval
https://glama.ai/mcp/servers/RudrenduPaul/agent-eval
NPM: https://www.npmjs.com/package/agent-regress-cli
PyPI: https://pypi.org/project/agent-regress-cli

We would love to hear your feedback and answer any technical questions on this topic. Let us know what you think!