technical
Refactoring Codebases at Scale

Refactoring Codebases at Scale

How one engineer rewrote our entire platform in a single day with AI agents by treating the codebase as a system to reshape, and the six guardrails that kept it from falling apart.

MA
Matthieu Jabbour
CTO

Table of Contents

  1. Introduction
  2. From lines of code to a system
  3. A test harness that only knows the contract
  4. Git as an unbypassable gatekeeper
  5. End-to-end tests have the final word
  6. From documentation to policy
  7. An opinionated framework to remove ambiguity
  8. Fixing prompts, never code
  9. From a one-off rewrite to everyday work
  10. Conclusion

Introduction

A week ago, we rewrote our codebase. All of it: roughly 250,000 lines of code, spread across 10 repositories, in a single day. The whole initiative cost us about 1.6 billion tokens, one engineer’s full-time day, and the work of some 30 agents running in parallel.

The goal wasn’t to change what the platform does. It was to improve its design and architecture: simplify it, make it more maintainable and extensible, and get rid of patterns we had stopped wanting to see. Along the way, it also fixed about 50 bugs and edge cases.

Back in April, when we wrote about handing our backlog to agents, we listed “harder tasks” as the next frontier. Agents were great at well-scoped tickets, but on large cross-cutting refactors their context got compressed over long sessions and the quality slowly drifted. A full rewrite is the hardest version of that problem you can pick. It touches everything at once, and nothing can be validated in isolation.

Two things made it work. The first is the models themselves: Opus 5.5 and GPT 5.6 Sol have reached a level of reliability we wouldn’t have bet on a year ago, and without that, no amount of process would have held up at this scale. The second, which this post is about, is a change in how we think about code. Once agents can produce code faster than any human can read it, reasoning file by file and function by function stops scaling. You have to start reasoning about the codebase the way an architect reasons about a building: as one system, governed by a handful of rules, made of a material you can reshape.

This post walks through the six guardrails that let us do that without breaking production.

From lines of code to a system

For most of the history of software, code was expensive to produce and cheap to read. So we optimized for the unit we could hold in our heads: a function, a class, a file. Code review, unit tests, refactoring tools, all of them were built around that granularity.

Agents flip the economics. Producing 10,000 lines is now a matter of minutes, while reviewing 10,000 lines is still a matter of days. At 250,000 lines, line-by-line review is simply off the table (at a generous 1,000 lines reviewed per hour, that’s 250 hours of reading for a single pass).

So the question changes. It’s no longer “is this function correct?” but “what constraints, if enforced everywhere, guarantee that whatever the agents produce is correct?”. You stop editing the material directly and start shaping the mold.

Concretely, the mold had six parts:

What must hold The contract
  1. 1 Contract tests The platform does what it does today
  2. 2 Git lock Nobody touches the contract
  3. 3 End-to-end tests It also works against the real world
How to get there The shape
  1. 4 Handbook The policy agents must follow
  2. 5 Framework One obvious way to do each thing
  3. 6 Prompts, not code Humans steer the system, not the lines

The first group defines what must stay true. The second shapes how the code looks once it’s rewritten. Let’s go through them in order.

A test harness that only knows the contract

Before rewriting anything, we completed our automated test suite (unit tests and end-to-end tests) and pushed it to 100% coverage on every repository and service. We weren’t starting from scratch: coverage averaged around 60%, and some repositories were already close to 100%. Still, this is where most of our time went, carefully reviewing the existing tests and writing the missing ones.

That sounds like a classic precondition for any refactor, and it is. The catch is that most test suites would have been useless for this job, because of how they’re written.

The problem with “unit” tests

The textbook unit test isolates one function or method, mocks every collaborator around it, and asserts on how it was called. Martin Fowler calls this the mockist style, as opposed to the classical style where real objects are used wherever possible. Mockist tests are precise, but they encode the implementation: which private helper gets called, in what order, with what intermediate values.

That is exactly the knowledge a rewrite destroys. Rename a method, extract a helper, merge two classes, and hundreds of tests go red while the product behaves identically. At that point you can’t tell a real regression from noise, and you face a choice that kills the whole idea: rewrite the tests along with the code. Kent Beck lists this property as one of his Test Desiderata: tests should be structure-insensitive, meaning they should not change their result if the structure of the code changes.

Testing the public contract, and nothing else

So we changed our definition of a unit test. A test calls the code the way its real callers do, through its public API, and checks only what those callers can observe. Everything in between runs for real: internal methods, helpers, formatting, validation. The only things we replace are the dependencies that leave the process, like database clients, REST API calls, TCP connections, or message queues.

This is the diagram that opens the testing chapter of our internal handbook:

Your test engine.create() Asserts on Returned value Thrown error Calls to mocks
Your code (all real)
  1. Public method
  2. Internal methods
  3. Helpers
  4. Collaborators
Mocks stand in here
The outside world
  • PostgreSQL
  • Slack
  • HTTP APIs
  • Kafka
Tests go through the public API and everything inside the process runs for real. The only thing replaced is what leaves the process.

The scissors are the whole method. From there, the rule for writing assertions fits in two lines:

  • If the code is pure (same input, same output, no side effect), test the output for a given input. Nothing else.
  • If the code has side effects (database writes, outgoing HTTP calls, notifications), also check that those external calls happen, with the right parameters. Still nothing else.

Here is what it looks like on a small ticket engine from our handbook, which formats a title, computes a due date and alerts the team on urgent tickets:

vi.mock('scripts/services/Notifier');
vi.mock('scripts/services/DatabaseClient');

test('opens an urgent ticket and alerts the team', async ({ engine, databaseClient, notifier }) => {
  const ticket = await engine.create('  Coffee machine   is leaking ', 'HIGH');

  // Observable output.
  expect(ticket).toEqual({
    id: expect.any(String) as string,
    priority: 'HIGH',
    status: 'OPEN',
    title: 'Coffee machine is leaking',
    dueAt: new Date('2025-01-01T13:00:00.000Z'),
  });

  // Side effects leaving the process.
  expect(databaseClient.insert).toHaveBeenCalledWith(ticket);
  expect(notifier.alert).toHaveBeenCalledWith('Urgent ticket: Coffee machine is leaking');
});

Notice what’s missing. There’s no assertion on the protected formatTitle method, nor on the computeDueDate helper it relies on. Both run for real, both are covered, and both can be renamed, inlined, or rewritten from scratch without this test ever noticing. A nice side effect is that one call through a public API exercises a lot of code at once, so 100% coverage comes with far fewer tests than the mockist approach would need.

Once this harness was in place, it captured every feature currently specified and shipped. From there the deal was simple: anything in the codebase can change, as long as the contract holds.

Git as an unbypassable gatekeeper

A harness only protects you if nobody tampers with it, and agents are not always trustworthy on that front. Even with today’s models, they hallucinate and drift off track, especially on long sessions. When a test goes red, the path of least resistance for an agent is sometimes to “fix” the test rather than the code. We had seen it happen on smaller tasks before.

On a rewrite of this size, a single agent quietly updating both a module and its tests would be a nightmare. The suite would still be green, and we would have lost any way of knowing whether behavior had regressed.

So we made the test files read-only by convention, and enforced by Git. On every iteration, the gate was the same three checks: run the unit tests, run the linter, and verify with git status / git diff that no test file had been modified. We ran it as a script, which boiled down to this:

#!/usr/bin/env bash
# Simplified version of the per-iteration gate.
set -euo pipefail

yarn lint
yarn test

# Any change to a test file or a mock fails the iteration.
if git status --porcelain -- '**/__tests__/**' '**/__mocks__/**' '*.test.ts' '*.test.tsx' | grep -q .; then
  echo "Test files were modified. Rejecting this iteration."
  git status --short -- '**/__tests__/**' '**/__mocks__/**'
  exit 1
fi

The mechanism is deliberately simple. Git is deterministic, cheap, and doesn’t hallucinate. Asking an agent “did you modify any test?” gives you an opinion, while asking Git gives you a fact. That check is what turned “the tests are green” into a statement we could actually trust.

End-to-end tests have the final word

Our unit tests have one deliberate blind spot: everything beyond the scissors is mocked. They prove that the code calls the database client with the right parameters, but not that the database accepts the query. A rewritten SQL statement, a changed serialization format, or a subtly different HTTP header can all pass unit tests and fail in production.

That’s what the end-to-end suite is for. It runs the platform in real conditions, with an actual database, actual APIs and actual services talking to each other, and it gets the last word. It covers a few hundred scenarios, focused on the most critical flows only, and its test files are locked by the same Git gate as the unit tests. If the unit tests are green and the end-to-end tests are red, the iteration is rejected.

The two layers complement each other neatly. Unit tests are fast and pinpoint where a contract broke; end-to-end tests are slower but catch the class of bug that only exists at the seams between components and the outside world.

This extra layer is what turned “probably safe” into real confidence. With unit tests alone, we would have known the code still honored its contract. With end-to-end tests on top, we also knew the rewritten platform wouldn’t break production or quietly introduce regressions. The results bore that out: once the rewrite was deployed to our QA environment, only two small regressions surfaced, and both were fixed within minutes.

From documentation to policy

With the top half of the mold in place, we knew the platform would keep working. What it would look like after the rewrite was a different question.

Agents make thousands of small design decisions per hour: where to put a file, how to name a function, how to handle an error, which abstraction to reach for. Left alone, they will make them inconsistently. Two sessions on the same prompt will happily produce two different structures.

The rule we followed was strict. Every time we noticed an architectural choice, pattern, or convention we disagreed with, we left the code alone and updated the handbook instead. Then we had agents review the entire codebase against the updated handbook.

Our MerciYanis Handbook is a 100% markdown repository covering architecture, principles, best practices, security, scalability, infrastructure, and step-by-step recipes like the testing guide quoted above. During the rewrite it stopped being documentation and became something closer to policy: a set of rules the codebase must satisfy, which agents enforce.

The workflow, every time a bad pattern showed up, went like this:

  1. Spot the pattern (an anti-pattern, an inconsistency, a convention we don’t like).
  2. Generalize it: is this a one-off, or a family of problems across the codebase?
  3. Write the rule down in the relevant handbook chapter, with the reasoning and a reference example.
  4. Have agents sweep the whole codebase against the new rule.
  5. Run the Git gate and the end-to-end suite.

The difference with fixing the instance by hand is enormous. A manual fix corrects one occurrence and teaches nothing to the next agent session. A handbook rule corrects every occurrence, including the ones that haven’t been written yet.

In practice, the handbook was already fairly complete when we started, so this mostly meant filling gaps. We added a handful of chapters along the way: observability patterns, pre-QA code polish, code documentation, and file splitting.

An opinionated framework to remove ambiguity

A handbook can only say so much. The other half of steering agents is shrinking the space of valid answers in the first place.

That’s what an opinionated framework does. It comes with strong conventions about file structure, naming, wiring, and the architectural patterns you must follow for things to work at all. For a human team, that can feel restrictive. For agents, it’s exactly what we want: a well-marked trail with one obvious way of doing each thing, instead of an open field.

We had committed to Perseid long before agents entered the picture, and it paid off here once more. When the framework already dictates where a controller goes, what a repository looks like, and how a service is wired, the agent doesn’t have to invent any of it, and neither does the next agent working on the next file. Consistency across 250,000 lines stops being something you review for and becomes something you get by construction.

Our handbook has a dedicated Perseid chapter for that reason, so the framework’s conventions and our own policy live side by side, and agents read both before touching the code.

Fixing prompts, never code

The last principle is the one that holds all the others together, and the hardest one to stick to.

When something in the output is wrong, the reflex of any experienced engineer is to open the file and fix it. It’s faster, it’s satisfying, and on a rewrite of this size it’s a trap. A hand-edit fixes one occurrence and leaves the system that produced it unchanged, so the same mistake comes back on the next run, in another file.

So we treated every defect as a signal about the system. Was the handbook unclear? Was a rule missing? Was a slash command giving the agent the wrong context, or the wrong order of operations? We fixed that, and let the agents regenerate the code.

The way we see it now, the code is the output of a system made of three inputs: the contract (tests), the policy (handbook and framework) and the process (prompts and commands). Our job during that day was to adjust those three inputs until the output looked exactly the way we wanted. It’s a strange feeling at first, a bit like a sculptor who’s only allowed to touch the mold, but it is what makes the approach scale.

From a one-off rewrite to everyday work

The rewrite was a one-shot effort: a single, well-delimited day with a clear goal. The harder question is what comes after it. How do we bring the same principles into our daily operations, where work arrives as a steady stream of features and fixes rather than one big push?

We already know where the pressure lies, because we feel it every day. Pull request review has become the new bottleneck. Agents open PRs faster than we can read them, and the time a PR spends waiting for a human now weighs more on our velocity than the time it takes to write it. It’s the same imbalance we described at the start of this post, only spread across hundreds of small changes instead of one large one.

The rewrite suggests a way out. During that day, we didn’t review 250,000 lines one by one. We reviewed the codebase from a higher vantage point, enough to make sure everything was built as intended, and spent most of our attention on the system that produced it: the contract, the policy, and the process. Applying that lesson to daily work means moving human attention away from the implementation and toward the few places where it really matters:

  • The contract. In feature work, tests do legitimately change, so the Git lock can’t simply forbid it. But it can flag it. A PR that leaves the tests untouched is a refactor that the harness already vouches for. A PR that changes them is a change in behavior, and that diff, much smaller than the implementation, is where a human should look first.
  • The policy. Every review comment that points out a pattern is a missing handbook rule. Writing that rule down, rather than leaving the comment, means the next PR doesn’t need the same remark.
  • The process. When a PR comes out wrong, the fix belongs in the prompts and commands that produced it, so that the next one comes out right without anyone having to catch it.

We haven’t figured all of this out yet, and turning review into something that scales with agent output is the next problem we’re working on. We’ll share what we learn along the way.

Conclusion

Rewriting 250,000 lines in a day wasn’t about typing faster, or even about letting agents type for us. Models reliable enough to be trusted with that much code were a prerequisite, but they weren’t sufficient on their own. The rest came from treating the codebase as a system you shape through constraints: a contract that defines what must stay true, a lock that keeps that contract honest, real-world tests that have the final word, and a policy (handbook, framework, prompts) that defines what good looks like.

Once those pieces are in place, the code itself becomes a malleable material. It’s still the thing we ship, but it’s no longer the thing we edit.

If you’re attempting something similar, or think we got something wrong, we’d love to hear about it at engineering@merciyanis.com.