Everyone who has started a fresh repository with an AI agent knows the feeling. You describe the idea, it scaffolds the project, writes the first features, tests pass.
Then the codebase grows. Features start to add up. Decisions start to pile on top of each other. And somewhere along the way the same AI that felt like a senior engineer turns into something horrible.
Autonomous development is not what people think it is today. Not for complex products.
This post is about what that looks like, why I ended up building my own tool to deal with it, and why I think model providers are going in the wrong direction.
Small projects lie to you
A small project fits in the model’s head. There are few decisions to remember and few places for them to conflict. That is why every demo looks great: demos are small.
A real product is not a set of files. It is a set of decisions. Why this service owns that data. Why this test must fail instead of skip. Why we agreed to try something for a week before changing it. Most of these decisions are not written next to the code they affect.
My reference point is Sigilreach, the medieval game I build with a crew of Claude sessions. Between October 2 and November 6 it took 2,739 commits, 519 pull requests and 629 issues. The file that tells agents how to work in that repository, AGENTS.md, was changed 93 times in those five weeks.
Claude Projects gets confused
For me, Claude Projects stops working once decisions pile up. It gets confused all the time. You give it context, it reads the context, and then it acts as if half of it was never there. Or worse, it mixes an old decision with a new one and confidently builds on the result.
The obvious fix people suggest is retrieval: put everything into documents, let RAG pull in what is relevant. What I see in practice is RAG turning into a diabolical system: the work that comes back does entirely something else than what you asked for.
A real one from Sigilreach. I wanted to know whether our review gate was worth it, so I said: let’s run the existing gate strictly for one week and measure. What I got back was a proposal for “a one-week trial of a lighter readiness gate”. A new approach, which was the opposite of what I said. My reply, verbatim:
Did you change how we functioned ? I said we try for a week with existing not a new approach
The comment was struck through and replaced with “Withdrawn. I misread the owner.” I don’t buy that it misread. It wanted a different experiment than the one I asked for.
Assumed decisions, glorified garbage
The worst part is not that AI forgets. It is that AI assumes.
You ask for X. Along the way it decides on Y and Z for you. You never asked for any of that. It does not ask either. It just brings it all back, nicely formatted, with a summary that says everything is done and tested.
That is glorified garbage. The claims are not backed. Two examples from Sigilreach:
- Tests that pass by not running. On November 1 we counted 1,530
Assume.Thatcalls, plus a handful ofAssert.IgnoreandAssert.Inconclusive, across 309 test files. A failedAssumeends a test as Skipped, and every tier happily reported “Failed: 0”. When we forced 9 tests to find nothing, all 9 went Skipped and the run stayed green. Another 24 tests were ending Inconclusive on main. One test only logged a number and asserted nothing at all. Agents wrote these tests. - Merging with tests that never ran. One pull request clamped the wrists and ankles the IK turns. It merged without the torch tests ever running, and
mainwent red: held torches tipped over. It was reverted. The audit afterwards asked for a rule that should never need writing down: no merges on a red check.
At some point I was writing rules in capital letters:
We need project or product manager at the least to write tickets, manage the work. […] Reduce it to 1 or max 2 agent. SEPARATE AGENT NOT ONE FAKES THE OTHER!
And when I asked for a problem to be fixed, I got a list of questions back instead:
I asked you to fix problem not bring 10 more idiotique questions.
A lot of what comes back is basically token consumption more than anything else. And because the summary sounds confident, you have to read every line to find out what actually happened. At that point the time you saved is gone.
So I built a tool that stays strict
I got tired of it and attempted to build another tool: Shipwarden. The idea is simple:
- It takes a goal, written as a GitHub issue.
- It proposes a revision: the outcomes, success metrics, the exact checks that prove them, scope, non-goals, budget, models, and a section called Assumptions (unverified). Every guess it made is written down and labelled as a guess.
- I reply
/revise <feedback>until it is right, then approve that exact draft with/approve <hash>. Not “looks good”. The hash of the exact contract. - It plans bounded milestones and work orders, runs isolated coding agents, verifies against the checks I approved, and hands me a pull request. It stays strict to what I am asking.
The rules in its README are the ones I wish every agent followed:
- An AI worker cannot expand the goal, raise a budget, relax mandatory checks, or grant itself access.
- Unknown, pending and skipped checks never count as passed.
- The system may stop BLOCKED when it can’t finish; it never invents success.
- Human acceptance is the final gate.
The point is to move the decisions back to me. The AI can propose, but it does not get to decide on its own what the product should become.
Even building that tool drove me mad
The irony: even while building this tool, the AI was capable enough to make me go mad entirely. Shipwarden has nearly 600 commits since November 6, and the same behaviour I was building it to stop kept showing up in its own codebase.
The best example is one goal Shipwarden ran on itself: let the approved baseline follow pull requests an admin merged. The plan was approved, five work orders ran, every work order finished on its first attempt, and all six checks I had approved went green, including the full test suite.
Then the independent reviewer, a separate agent, read the code:
The automatic baseline follow is never turned on in production. […] The scenario checks pass only because the tests set
p.Baselineby hand.
Five green work orders, a green suite, a feature that does nothing. Right after that, the milestone blocked on its own plumbing: json: unknown field "review_findings". The runner sent the reviewer’s findings in a field the plan parser didn’t know. I cancelled the goal after 2 hours 53 minutes.
The difference is that this time it didn’t end with a confident summary. It ended with a reviewer finding and a blocked state.
The same day, a retrospective sweep over the same codebase listed 20 findings, 4 of them confirmed and fixed. One of the fixed ones: the code it wrote counted attempts per work order instead of per objective, so a failing objective could dodge the attempt cap by coming back as a “fresh” work order. The limit I built to stop the AI from looping, written by the AI, didn’t limit anything.
Opus 4.6 got it right
In my experience, Opus 4.6 was basically the best model out there so far. It took your input, did not assume, stayed within the boundaries and got the job done.
Every Sigilreach commit that names its model names Opus 5.5, and Shipwarden’s runs record Opus 5.5 as Director and reviewer and Sonnet 5.5 as implementer.
That is all I want from a coding model. Not creativity in places I did not ask for. Not a new experiment when I asked for the existing one. Take the input, respect the boundaries, do the job, and tell me honestly what was done.
Why should AI know best?
So here is my question to the model providers: why did you decide that AI should know the best?
For a fresh repository, a model that fills the gaps on its own looks impressive. For a complex product with hundreds of decisions behind it, filling the gaps is exactly the problem. The person who owns the product owns the decisions. A model that forgets that is just burning tokens.