About Harness Notes

This project delves into the intricacies of production agent harnesses, exploring how twelve different teams manage twelve distinct jobs. The core idea is that while the AI models themselves might be similar, the surrounding code is what truly differentiates an agent's performance in terms of cost, safety, and reliability. The author is documenting their journey of understanding and implementing these agent harnesses, one job at a time.

• Comparative Analysis: The project compares twelve production agent harnesses, highlighting that differences lie more in the implementation of checks rather than the core loop execution. Key factors influencing performance are the ease of catching mistakes and the cost of missed errors, which ultimately lead to lower costs per successful task.

• Personal Agent Development: The author is building and scoring their own drafting agent, measuring its performance based on cost per accepted draft. This process involves setting clear guardrails and pass marks before the agent runs, and any changes made are documented with supporting data.

• Subscription Service: Readers can opt to receive future posts via email, with a system in place to handle subscriptions and confirmations.

The project includes an agenda for the ongoing work, a build log to track progress, and a section for corrections. It is a public, in-progress documentation of building and evaluating AI agents.