<p dir="ltr">It&rsquo;s not everyday an academic study on machine learning garners millions of views on X in just a few days. &ldquo;<a href="https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf">Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity</a>&rdquo; from Model Evaluation &amp; Threat Research (METR) challenges a widely accepted truth that AI makes developers more productive.&nbsp;</p>
<p dir="ltr">In the paper, the authors reveal a finding shocking to many: when experienced developers were given access to state-of-the-art AI tools from early 2025, like Claude 3.7 Sonnet and Cursor Pro, they were 19% slower than their counterparts. Even more surprisingly, those same developers felt as though they had been sped up by 20%. This disconnect wasn't limited to the participants; outside economics and machine learning experts also dramatically overestimated the tools' impact, predicting an average speedup of nearly 40%.</p>
<p dir="ltr">Is it possible that the work just felt faster because it was more enjoyable? Or that the developers were simply taking time to test the limits of new tools?&nbsp;The fuller story exists between the clock and the human experience. The paper offers us an opportunity to examine where we&rsquo;re at with AI coding tools, the kinds of problems we use them to solve, how they are used, and who is able to use them.</p>
<h2 dir="ltr">Reasons for the Slowdown</h2>
<p dir="ltr">The study's results suggest the slowdown isn't a single issue, but stems from several factors. Among them, three major themes of friction stand out:&nbsp;</p>
<ol>
<li dir="ltr" aria-level="1">
<h3 dir="ltr" role="presentation">The Problem of Context</h3>
</li>
</ol>
<p class="whitespace-normal break-words">Think of it as an 80/20 problem: AI can handle the initial 80% of coding tasks reasonably well, but struggles with the final 20% where deep context matters most. This was especially apparent with the expert developers in the study, who averaged five years of experience and 1,500 commits on massive, mature codebases.</p>
<p class="whitespace-normal break-words">Developers reported that AI often behaved like an inexperienced team member struggling with backwards compatibility or proposing edits in the wrong locations. The numbers tell the story: they spent 9% of their time reviewing and cleaning AI outputs, plus another 4% waiting for generations. The problem was worst on tasks requiring institutional memory, because that knowledge represents rich context built from years of human experience.</p>
<p class="whitespace-normal break-words">This means the study tested AI in perhaps its most challenging scenario&nbsp;where human expertise and context are at their peak. The findings may not apply as broadly to junior developers or greenfield projects, though this friction will likely decrease as AI gets better at handling larger contexts.</p>
<ol start="2">
<li dir="ltr" aria-level="1">
<h3 dir="ltr" role="presentation">New Tools = Moving Targets</h3>
</li>
</ol>
<p dir="ltr">Second, effectively managing these tools is a new skill that many are still getting good at. For example, it is a learned skill to know precisely when to abandon a failing dialogue and finish the work manually. Mastering this is difficult because the tools themselves are a moving target, constantly evolving. Imagine a carpenter having to learn how to use a new hammer every few weeks.&nbsp;</p>
<ol start="3">
<li dir="ltr" aria-level="1">
<h3 dir="ltr" role="presentation">Structural Bottleneck</h3>
</li>
</ol>
<p dir="ltr">Finally, the tools themselves impose a structural bottleneck. While a human developer mentally juggles multiple dependencies and plans their next steps while working on a piece of code, interacting with most AI assistants forces them into a rigid, serial workflow: one instruction, one response. This doesn't align with the way experts think.&nbsp;</p>
<p dir="ltr">However, this is already beginning to change, with a potential shift towards a more asynchronous, agent-based workflow. This is a different model than the one tested in the paper, and while it's an interesting evolution, it isn't yet clear if it&rsquo;s more effective at reducing development time, a question future studies may need to address. In this emerging future, we might see a hybrid approach where developers choose their tools based on the complexity of the task using a fully autonomous agent for some things and hands-on AI assistance for others, perhaps even using one to fix the other's work.&nbsp;</p>
<h2 dir="ltr">So, Which Developers are Empowered by AI?</h2>
<p dir="ltr">While the study's focus on expert productivity is valuable, looking beyond this specific research suggests it tells only a partial story. For existing developers, AI's greatest impact may not yet be on finishing complex tasks, but on dramatically speeding up the early stages of a project, allowing them to jump from inspiration to a working prototype much faster. But perhaps its most transformative effect is opening the door to coding for an entirely new population. This is happening in two key ways. First, it lowers the "activation energy" for managers and executives strapped for time, who can now build tools in short bursts. Second, it's empowering a wave of vibe coders including product managers, designers, or other kinds of non-engineers to build their own tools and programs with minimal engineering resources. In both cases, we are seeing entirely new groups of people do things they would never have been able to do before.</p>
<h2 dir="ltr">A Clear-Eyed View of the Future</h2>
<p dir="ltr">On one hand, the study is a valuable, data-driven look at the friction AI can create for experts on complex, context-heavy tasks. On the other, the technology is simultaneously a powerful democratizing force, unlocking entirely new value by empowering new populations of coders. This comes with its own forward-looking questions. A heavy reliance on vibe coding may not engage traditional coding muscles, potentially leading to an atrophy of deeper engineering skills over time.&nbsp;</p>
<p dir="ltr">The paper's authors, however, caution against overgeneralizing. The impact could be entirely different, and perhaps positive, for junior developers or those starting on new, greenfield projects.&nbsp;</p>
<p dir="ltr">With that in mind, perhaps the most telling clue to the future is also the most paradoxical finding in the entire study: despite being measurably slower, 69% of the developers continued using the AI tool after the experiment ended. That single data point proves the ultimate thesis. The value of AI cannot be measured by the clock alone; it&rsquo;s a complex equation of speed, context, enjoyment, and empowerment. Understanding that complexity is the key to navigating the future of work.</p>
<p>&nbsp;</p>

A METR study found AI slowed devs by 19%, but they felt 20% faster. We unpack this paradox of productivity, skill, & AI.