In 2025, METR ran the study everybody had been arguing about without data. Sixteen experienced open-source developers, 246 real issues, in repositories they had worked in for about five years. Half the tasks randomised to allow AI tools, half not.
The developers expected to be 24% faster. Afterwards, they believed they had been 20% faster. The stopwatch said they were 19% slower.
The gap between the second bar and the third is the most important number in AI-assisted engineering, and it is not about the models at all. It is about our ability to tell whether they are helping.
Two caveats before anyone quotes this as a verdict. It is sixteen developers, which is small. And METR themselves now label the result historical: it measured early-2025 tools and early-2025 habits, and both have moved. I am not citing it as proof that AI makes engineers slower. I am citing it because it is the only careful measurement of the thing we are all confidently asserting, and it came out backwards.
Why the gap exists
The subjective experience of using a coding model is that the waiting stops. You do not sit there deciding how to name the thing, or reading documentation for the argument order, or writing the boilerplate you have written a hundred times. Output appears immediately. It feels fast because it is fast, in the parts you were paying attention to.
The time goes somewhere else. Reading code you did not write to decide whether it is right. Discovering it is subtly wrong in a way that would have been obvious if you had written it. Re-prompting. Reverting. Doing it yourself, but now with the model’s framing stuck in your head.
None of that registers as work in the way that staring at an empty file does. So it does not get counted, and the estimate comes out at +20% while the clock says −19%.
Where the line actually falls
The pattern in my own work, and the one the study is consistent with: the advantage is largest exactly where your own knowledge is smallest.
- Unfamiliar territory: large gain. A language you use twice a year, an API you have never touched, a config format with bad documentation. The model has read more of it than you have.
- Well-specified and mechanical: real gain. Translate this schema, write these tests from this spec, refactor this pattern across forty files. Clear inputs, checkable outputs.
- Code you know intimately: no gain, sometimes negative. This is the METR condition: mature repositories the developers had spent years in. Their internal model of the codebase was better than anything in the context window, and describing what they wanted cost more than doing it.
- Architecture and long-horizon design: no. Not because models cannot reason, but because the constraints that matter are unwritten: what the team can maintain, what the last migration cost, which dependency the company is trying to get off. None of that is in the repository.
The uncomfortable implication is that the tools help most where you are least equipped to check them. That is not a reason to avoid them. It is a reason to be specific about which mode you are in.
The autopilot comparison, done properly
The aviation analogy gets used constantly and usually badly, so here it is with the details right, because the details are the whole lesson.
Qantas Flight 72, October 2008. A faulty air data unit fed the flight computers garbage, which commanded two violent uncommanded pitch-downs. The crew disconnected the automation, flew the aircraft manually, and landed at Learmonth. Everyone survived. Automation failed in a way its designers had not anticipated, and skilled humans caught it.
Air France 447, June 2009. Iced pitot tubes gave inconsistent airspeed, the autopilot disconnected as designed and handed control back, and the crew, suddenly manual, at night, over the Atlantic, with no clear picture of what had happened, held the nose up and stalled the aircraft into the ocean.
Same category of trigger, opposite outcome. The difference was not whether the automation failed; in both cases it did. The difference was whether the humans still had the skill and the situational picture to take over in the seconds after it handed control back.
That is the risk worth naming for engineering. Not that AI writes bad code. That we stop maintaining the ability to evaluate it, and find out which of those two flights we are on at the worst possible moment.
What to actually do
Three things, in decreasing order of how confident I am about them.
Measure it, at least once. The METR result exists because someone put a clock on it. You do not need a randomised trial: take ten tasks, do five with the tools and five without, and write down the times before you form an opinion. If your estimate matches your clock, good. If it does not, you have learned the single most useful thing in this post.
Know which mode you are in. Unfamiliar and mechanical: lean on it hard. Deeply familiar and load-bearing: write it yourself and use the model to review. The failure is not using the tool, it is using it identically in both cases.
Keep the skill you are automating. Read the generated code as if you were reviewing a colleague’s pull request, because that is exactly what it is, from a colleague who is confident, fast, widely-read, and has no idea what your system is for. Review is the skill that stays valuable in every version of this future, and it is also the one that atrophies quietly if you let it.
The best engineers in five years will not be the ones who generate the most code. They will be the ones who can still tell, quickly and reliably, whether the code in front of them is right.