ContextOps

In 2025, METR — an AI safety and capability research organization — ran a rigorous randomized controlled trial. Sixteen experienced open-source developers worked on 246 real-world tasks, each randomly assigned to either use AI coding tools freely or not at all. The result was counterintuitive: developers using AI tools were 19% slower on complex tasks. Before the study, those same developers predicted AI would make them 24% faster. After completing the experiment — still believing they had gone faster — their subjective confidence remained completely unshaken. ...

Series Orientation: This article is Part 6 of the AI Code Review & Vibe Coding series, looking at team governance and developer career paths. For the preceding security chapters, see Part 5 — AI Code Security. As highlighted earlier in this series, the METR study (2025) revealed a striking paradox: experienced developers using AI tools were actually 19% slower on complex real-world tasks, even while believing they were 24% faster. ...