The Hidden Cost of AI-Generated Code Is Reviewing It

The reviewer has to reconstruct the problem

A pull request is a proposed change to a system people already depend on.

To review it properly, I need more than a summary of the files that changed. I need to understand the problem, the constraints, and the behavior we intend to introduce.

Why this approach? Which assumptions does it depend on? What happens to existing users? What would make us roll it back?

The code contains some of those answers. It rarely contains all of them.

When the author understands the change, we can have a useful conversation. They can explain a decision, point out uncertainty, and tell me which alternatives they considered.

When they don’t, the reviewer inherits the investigation.

Now I’m figuring out what the implementation does, why it might have been written that way, and whether any of it matches the original requirement.

That is a substantial amount of work to hide behind “ready for review.”

A convincing explanation is still something to check

AI can produce a detailed pull-request description. That can be useful.

But the quality of the explanation and the correctness of the implementation are separate things.

A description might say a change prevents duplicate processing. The code might check whether an event exists, then insert it.

That looks reasonable until two requests arrive together.

Both requests check. Neither finds the event. Both proceed.

The missing detail is the concurrency behavior. Does the database enforce uniqueness? Is the operation atomic? What happens when the second request loses the race?

Those are the questions a reviewer needs to ask.

If the author has already investigated them, the review can focus on checking the reasoning and implementation. If the author simply repeats the generated description, someone else has to do that investigation.

Fluent documentation helps communicate understanding. It doesn’t establish that understanding exists.

Passing tests doesn’t settle the requirement

Tests matter. I want them.

I also want to know what they prove.

Suppose an agent writes an implementation and generates tests from the same interpretation of a ticket. Both can agree with each other while missing what the system actually needs to do.

A retry test might verify that a failed request runs again. It may never check what happens when the original request succeeded but its response was lost.

The test passes. The duplicate charge is still possible.

This problem existed before AI. What changes is how quickly we can produce an implementation and a convincing collection of tests around it.

So part of review is checking the test boundary:

• Which requirements came from the actual problem?
• Which failure cases were considered?
• What existing behavior must remain intact?
• What evidence would reveal that our assumptions are wrong?

More tests can improve coverage. Their value depends on the behavior they examine.

The queue can move without the team getting faster

Imagine a team where five engineers start opening twice as many pull requests.

Review capacity stays roughly the same.

What would we expect to happen?

The queue might grow. Reviewers might switch between more changes. Authors might wait longer for feedback. By the time a change is reviewed, the surrounding code may have moved.

Or the team might protect turnaround time by reviewing less deeply.

Neither outcome is inevitable. But both are plausible enough that I would check for them before declaring a productivity gain.

An individual can produce work faster while the team’s delivery process becomes more congested.

That is why I’m interested in the time from starting a change to operating it successfully in production. Generation time is one part of that journey.

A pull request waiting three days for someone to understand it is still unfinished work.

AI review can help, but it needs a defined role

Using AI to review generated code is a reasonable thing to explore.

An automated reviewer can provide another pass over a change, surface questions, and help people investigate potential problems.

The mistake would be treating that additional pass as a reason to stop asking what has actually been verified.

Did it have the relevant requirements? Could it inspect the surrounding system? Did it execute anything? Did a person check whether its findings were valid?

Microsoft’s account of using a coding agent in dotnet/runtime describes a process involving maintainer requests, review, and human intervention. It offers a concrete example of an organization evaluating agent contributions within an existing engineering workflow. It does not establish a universal result for every team or codebase.

That is the level at which I think we should evaluate these tools: specific work, specific conditions, observable results.

“Another model approved it” is information about the process. We still need to understand how much confidence that process deserves.

Microsoft’s engineering account

Authors should do the first serious review

My expectation is straightforward: before asking someone else to review a change, the author should have examined it themselves.

That includes generated code.

Read the diff. Follow the important execution paths. Check whether the tests address the requirement. Look for unrelated changes. Be explicit about what you haven’t verified.

You don’t have to know everything. You do need to distinguish what you understand from what you are assuming.

A useful handoff sounds like this:

“The database constraint prevents duplicate inserts. I tested simultaneous requests. The external notification can still fail after the transaction commits; that recovery path is outside this change.”

Now we can review the actual boundary.

Compare that with:

“The agent says it handles duplicates and all the tests pass.”

One gives the reviewer something concrete to assess. The other gives them homework.

Smaller changes make the reasoning easier to inspect

AI makes broad changes easier to generate. That doesn’t make broad changes easier to review.

A feature arrives with a refactor, renamed variables, a dependency update, and a new abstraction. Each part may look sensible on its own.

Together, they make it harder to identify which changes are necessary and which introduced a regression.

I’d rather separate changes where doing so preserves a coherent, testable result.

That isn’t a rule to split every task into tiny pull requests. Some work needs to be evaluated together.

The goal is to make the reasoning visible. A reviewer should be able to understand why each part is present without reconstructing several unrelated projects at once.

Measure what happens after generation

If I were evaluating AI adoption across a team, I would look beyond output counts.

I’d want to know:

• How long changes wait for review.
• How much revision they need before acceptance.
• How often reviewers discover missing requirements.
• How much follow-up work appears after release.
• Whether engineers can explain the systems they maintain.

These measures need context. A team doing a difficult migration will look different from a team making routine interface changes.

I wouldn’t turn them into individual rankings. That would encourage people to optimize the numbers instead of the work.

I would use them to ask a simpler question: where is effort accumulating, and are our tools helping us reduce it?

The responsibility still has to land somewhere

The first article in this series argued that code generation is an incomplete way to assess engineering work.

Review makes that argument concrete.

Someone has to connect the proposed implementation to the actual problem. Someone has to question the assumptions. Someone has to decide whether the evidence is strong enough to ship.

AI can help with those steps. Teams should examine where it helps and where it leaves work for someone else.

What I don’t want is an engineering culture where producing a change counts as success and understanding it becomes the next person’s problem.

If I put my name on the pull request, I should bring more than generated output.

I should bring an explanation I can defend, evidence someone else can inspect, and a clear account of what remains uncertain.

That is how I want us to judge progress.