Engineering Reliable AI Agents!

productHunt

devto

github

Sourav Nandy • Peerlist

Building Agent-Native B2A applications

A LangGraph agent fails on step 8. The trace shows exactly what broke. To reproduce it, you have to run the whole thing again: eight more calls to the model, about 30 seconds, and another 15 cents in API costs. If the failure came from the model giving an unusual response, you may not be able to reproduce it at all. You end up debugging something that will not hold still. We built Agent Observability to fix that. It is a Python library and command-line tool that records every network request your agent makes to a small local file, then replays those exact bytes offline, in about 0.93 milliseconds on average, compared to roughly 8,500 milliseconds for a real run against GPT-4o across 10 steps. Recording adds about 0.011% overhead, roughly 0.09 milliseconds per call to the model. The replayed result is identical, byte for byte, to what was recorded. You record once, and you can replay as many times as you want, for free. Record a run with <a target="_blank" rel="ugc" class="break-all link-break text-gray-500 hover:underline" href="https://www.producthunt.com/@tracer">@tracer</a>.instrument(record=True) around the call you want captured. Replay it with the replay() context manager, same code path, zero network, identical result. In CI, set AGENT_TRACE_NETWORK_GUARD=1 so any test that tries to make a real network call fails loudly right away, instead of quietly using up your API budget. Agent Observability is not meant to replace dashboards or evaluation tools. It solves one specific, earlier problem: reproducing a failed run without spending any more money on the model API, for any agent built on any Python HTTP client. Repo: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://github.com/RudrenduPaul/agent-observability">https://github.com/RudrenduPaul/agent-observability</a> MCP Servers: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://mcpservers.org/servers/rudrendupaul/agent-observability">https://mcpservers.org/servers/rudrendupaul/agent-observability</a> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://glama.ai/mcp/servers/RudrenduPaul/agent-observability">https://glama.ai/mcp/servers/RudrenduPaul/agent-observability</a> NPM: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://www.npmjs.com/package/agent-observability-trace-cli">https://www.npmjs.com/package/agent-observability-trace-cli</a> PyPI: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="http://pypi.org/project/agent-observability-trace-cli/">http://pypi.org/project/agent-observability-trace-cli</a> We would love to hear your feedback and answer any technical questions on this topic. Let us know what you think!

AgentObservability

The recurring failure mode is frustratingly common: an agent is updated, standard evals stay green, but performance noticeably degrades a week later. Two real-world examples catalyzed the development of this tool: - LangGraph PR #4486: Added node-level result caching, which quietly masked the repeated-sampling variance a regression test depended on, something a plain threshold check has no way to catch. - CrewAI PR #6134: Fixed a bug where file tools leaked absolute filesystem paths in responses. A standard quality scorer never flags this because it only checks if the answer is correct, not what is hidden in the payload. These examples are part of a larger validation pass encompassing 29 PRs and 239 rows across LangGraph, CrewAI, and the OpenAI Agents SDK. The full data breakdown is available in the <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="http://pr-analysis.md">pr-analysis.md</a> documentation on Github. Instead of grading a single response, agent-eval statistically compares two whole batches of runs. By running versions A and B 50 times each, the tool executes: - A Mann-Whitney U test (requiring no assumption that scores are normally distributed). - A bootstrap confidence interval to confirm if the shift is real. - Cohen's d to determine if the shift is large enough to actually matter. A p-value under 0.05 by itself is not enough to call it a regression. The effect size is required for definitive proof. As part of statistical guardrails, two limits are utilized to ensure data integrity. - Under 30 runs per version: Triggers a warning due to a drop in statistical power. - Under 10 runs: Returns INSUFFICIENT_DATA rather than guessing at an answer. Repo: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://github.com/RudrenduPaul/agent-eval">https://github.com/RudrenduPaul/agent-eval</a> MCP Servers: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://mcpservers.org/servers/rudrendupaul/agent-eval">https://mcpservers.org/servers/rudrendupaul/agent-eval</a> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://glama.ai/mcp/servers/RudrenduPaul/agent-eval">https://glama.ai/mcp/servers/RudrenduPaul/agent-eval</a> NPM: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://www.npmjs.com/package/agent-regress-cli">https://www.npmjs.com/package/agent-regress-cli</a> PyPI: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://pypi.org/project/agent-regress-cli">https://pypi.org/project/agent-regress-cli</a> We would love to hear your feedback and answer any technical questions on this topic. Let us know what you think!

AgentEval

We want to share a little background on why we built this project and the journey to getting it launched. We were inspired to build this because while working on distributed systems, we noticed a consistent struggle with how compute resources were being tracked and verified. we wanted a simpler and more transparent way to handle this tracking. The main problem we were trying to solve was that existing solutions for tracking compute were either too fragmented, lacked transparency, or were unnecessarily complex to integrate into standard workflows. ComputeLedger fixes that by providing a streamlined and verifiable ledger for compute operations. Our approach evolved significantly while working on this launch. Initially, we set out to build a basic logging tool. As we got deeper into the architecture, we realized the project needed a much more robust framework. That realization completely shifted how we built the core engine to make it scalable. Because we believe in building in public, ComputeLedger is entirely open source.Repo: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://github.com/RudrenduPaul/ComputeLedger">https://github.com/RudrenduPaul/ComputeLedger</a> MCP Servers: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://mcpservers.org/servers/rudrendupaul/computeledger">https://mcpservers.org/servers/rudrendupaul/computeledger</a> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://glama.ai/mcp/servers/RudrenduPaul/ComputeLedger">https://glama.ai/mcp/servers/RudrenduPaul/ComputeLedger</a> NPM: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://www.npmjs.com/package/computeledger-cli">https://www.npmjs.com/package/computeledger-cli</a> PyPI: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://pypi.org/project/computeledger-cli">https://pypi.org/project/computeledger-cli</a> We would love to hear your feedback and answer any technical questions on this topic. Let us know what you think!