Stop Switching Models. Fix This Instead.

I’ve been building with AI coding agents for the past few years across many products, ZeroCV, Round0 (an AI recruitment platform), Pinpoint BTC Flash Pool (a crypto precision market), Funnymoney (cross-chain meme coin trading), and Rankblitz — live quiz apps and many more and for most of that time, every time an agent task went sideways, my instinct was the same: the model isn’t good enough, let me try a better one.

Then I found this course called Learn Harness Engineering. 12 chapters, built on published research from Anthropic and OpenAI on what actually makes AI coding agents reliable. I read through all of it. And it reframed almost everything I thought I knew about working with agents.

This is what I took from it - the concepts that made me stop and think “oh, that’s exactly what was happening.”

The thing I most needed to hear: it’s probably not the model.

The course opens with this experiment Anthropic ran. Same prompt : build a 2D retro game editor. Same model :Claude Opus 4.5. Two runs.

First run: bare model, no support structure. 20 minutes. Core features broken.

Second run: full harness around it : a planner, a generator, a separate evaluator agent with actual browser-based click testing. 6 hours, $200. Fully playable game.

They didn’t change the model. Everything that changed was around it.

That hit differently reading it in the context of my own projects. Because I could see exactly where I’d been doing the bare-model version and wondering why I wasn’t getting better output. The course calls everything outside the model weights a harness — instructions, tools, environment, state, feedback loops and the core argument is: harness quality determines how much of the model’s capability actually lands.

I’d been optimizing the wrong thing.

There are only 5 reasons an agent task actually fails.

The course breaks this down into a clean diagnostic framework that I genuinely wish I’d had earlier:

1. Vague task spec : you asked for “a search feature” without saying what that means

2. Implicit conventions nowhere in code : rules that exist in your head but not in the repo

3. Broken dev environment : missing deps, wrong versions, agent burns context on setup errors

4. No verification method : nothing to run, so agent reads its own code and calls it done

5. Session state loss : every new session re-explores the same ground from scratch

Reading this, I could map almost every agent failure I’d had in the past year to one of these five. The Funnymoney one especially the agent kept writing token routing logic that worked on Solana but broke silently on Base. Chain-specific constraints were real. They just only existed in my head, not anywhere in the repo. That’s number two, exactly.

The framing shift the course suggests: before you touch the model, go through these five in order. Ask which one actually caused this failure. Once I started doing this, “the model isn’t good enough” almost never came up as the actual answer.

The AGENTS.md thing is real, but the bloat trap is real too.

Most people reading about AI coding agents have heard about adding an AGENTS.md or CLAUDE.md file to the repo. The course confirms it works but it also explains why a lot of people stop getting value from it after a few weeks.

The trap is that you start adding rules every time the agent makes a mistake. It works once. Then you add another rule. Then another. Three months later you have 500 lines and the agent is performing worse than when you started.

There’s actual research behind why: LLMs use information at the beginning and end of long texts much better than the middle just as humans naturally do. A critical constraint at line 300 of a 500-line file has a high probability of being ignored. I had this exact problem with Round0. The AI interviewer had a lot of moving parts: voice pipeline, evaluation logic, admin dashboard and every time the agent got something wrong I’d add a rule. My AGENTS.md grew to 400+ lines and the agent was confidently ignoring the important stuff because it was buried under months of accumulated rules.

The fix the course describes is a layered architecture. The entry file stays under 100 lines and acts as a router — project overview, quick start commands, 10–15 hard constraints, and links to topic docs. The topic docs live in docs/ or next to the relevant module, and the agent only reads them when they’re relevant. The example they give: a team split a 600-line file into an 80-line entry file and three topic docs- task success went from 45% to 72%, and security constraint compliance from 60% to 95%. Not because the rules changed. Because position in the file changed.

The session memory problem is worse than I thought.

This one I was already aware of as a problem, but the course made clear I was underestimating how much it was costing me.

On the live quiz project, Session 1 spent a significant chunk of time mapping the database schema, understanding the transaction model, figuring out how candidate and questions were structured. Session 2 had no memory of any of it. Re-explored the same ground. Re-inferred the same decisions. Fifteen minutes gone before any actual work happened.

The course makes the point that compaction strategies summarizing earlier context. preserve the what but lose the why. The next session sees the code but not the reasoning behind it. It then “optimizes” away deliberate decisions because it doesn’t know they were deliberate.

The solution they describe is treating agents like contractors whose memory resets every shift. Before clocking out, they document everything the next shift needs. Specifically: a PROGRESS.md tracking current state, completed work, in-progress items, blockers, and next steps and a DECISIONS.md logging major architectural choices with the reasoning and what was rejected.

The benchmark they suggest is rebuild cost under 3 minutes the time it takes a fresh session to reach “I know what to do next.” On projects without these files I was regularly at 10+ minutes. That’s not a model problem. That’s a missing handoff document.

WIP=1 sounds obvious and I still wasn’t doing it.

When I was building RankBlitz: a live quiz app with real-time rooms for admins and candidates. I gave the agent a broad task: “implement the live quiz room flow.” One hours later there was code everywhere, room creation logic, WebSocket handlers, candidate join flow, admin controls, scoring logic, leaderboard updates all half-touched. Working features end-to-end: zero.

The course has a name for this: overreach and it explains the math clearly. If the agent activates five tasks with a fixed context budget, each gets one-fifth of the reasoning available. When that fraction drops below what’s needed to complete any single task, nothing finishes. More code written, fewer features complete.

The rule they advocate is WIP=1, one feature active at a time, and you don’t start the next one until the current one passes end-to-end verification. They put the constraint directly in the instruction file. They also redefine “done” — not “code is written” but “verification command passes.” Every feature gets a specific verification command defined before work starts.

Anthropic’s data on this: agents using a single-next-step strategy showed a 37% higher task completion rate than agents given broad prompts. The number of lines of code generated is weakly negatively correlated with feature completion. I’ve seen this exact pattern enough times that this didn’t surprise me. it just gave me the vocabulary for it.

Agents grade themselves generously, and that’s a design problem.

This chapter was the most uncomfortable to read because I could see it in my own mental model. I’d been treating agent self-assessment as roughly trustworthy. The course makes the case that it isn’t, just as humans: our self evaluation of things are also very generous and most of the times wrong in estimates.

There’s research showing modern neural networks report confidence significantly higher than their actual accuracy. The agent saying “looks good to me” isn’t being dishonest, it genuinely believes that. But it generated the code, so it’s systemically inclined to trust the code. You’re asking the writer to be the editor.

The experiment they describe involved the same model running as both generator and evaluator, vs. a separate evaluator instance explicitly instructed to be critical, with a rubric and end-to-end test access. The separate evaluator caught things the generator-evaluating-itself consistently missed. The fix isn’t prompting the agent to “be more critical.” The fix is architectural: separating who builds from who checks.

The secondary point that I found really useful: unit tests consistently miss a whole class of bugs. Interface mismatches between components, state propagation errors across layers, resource leaks that span multiple components. The BTC Flash Pool WebSocket issue, Binance price feed working fine in isolation, TradingView chart working fine in isolation, silent state sync failure when they ran together. Zero unit tests caught it. One end-to-end run caught it in the first minute. The course argues that knowing an E2E test will run actually changes how agents write the code in the first place, they think about interfaces, not just the unit.

The entropy problem is real and it sneaks up on you.

The last few chapters of the course cover something I hadn’t seen named clearly before: without active cleanup discipline, AI-assisted codebases degrade in a specific and predictable way.

Agents copy patterns already present in the repo. including bad ones. One debug log left in becomes the pattern for the next agent to copy. One inconsistent naming convention gets reinforced. The entropy compounds.

The course shows week-by-week numbers on a 12-week project without cleanup discipline: build pass rate drops from 100% to 68%, test pass rate to 61%, session startup time to 60+ minutes. Same project with a cleanup routine: week 12 build pass rate 97%, tests 95%, startup 9 minutes.

The cleanup routine itself is simple. a session exit checklist: build passes, all tests pass (including pre-existing ones), feature list updated, no debug code remaining, startup path works for the next session. The key insight is making this a hard requirement for “done” rather than a nice-to-have. Session is not complete until all five are checked.

The course also makes a point I appreciated about the harness itself: as models improve, some components become unnecessary overhead. Anthropic removed a sprint-splitting mechanism from their production harness when a newer model could handle work decomposition on its own. The right move is periodically testing whether each component still earns its place: disable it, run your benchmark tasks, see if results degrade. If they don’t, remove it permanently.

The takeaway I keep coming back to.

What I got from this course, more than any specific technique, is a diagnostic mindset. Every agent failure maps to one of the five layers. Before you touch the model, identify which one. Usually it’s obvious once you ask the question.

The harness stuff - instruction files, progress tracking, verification commands, session exit checklists. none of it is magic. It’s the engineering discipline of creating an environment where the model’s existing capability can actually land. The model handles the reasoning. The harness handles everything else.

I’ve already started applying most of this. The instruction file split was the first thing. Progress and decisions tracking was the second. WIP=1 is the one I’m still actively building the habit for.

If you’re spending more time frustrated with your AI coding agent than productive with it, this course is worth reading. It explains more clearly than anything else I’ve found why the same model produces wildly different results in different setups and what you can actually do about it.

Course: Learn Harness Engineering based on published engineering research from Anthropic and OpenAI on long-running agents.