AI Coding Tools Slow Experienced Developers by 19 Percent
Model Evaluation and Threat Research found that experienced developers took 19% longer to complete tasks using AI assistants despite believing their productivity had increased.
A study by the nonprofit Model Evaluation and Threat Research (METR) found that experienced software developers using AI coding assistants took 19% longer to complete real-world tasks than those working without them. The randomized controlled trial tracked 16 seasoned developers across 246 tasks in mature open-source repositories averaging over one million lines of code. Participants primarily used Cursor Pro paired with Claude 3.5 or 3.7 Sonnet.
A significant perception gap emerged during the study. Developers predicted a 24% reduction in completion time before the experiment and estimated a 20% productivity increase afterward, despite the actual slowdown. Researchers attributed the decline to the friction of integrating probabilistic AI suggestions into deterministic workflows, the time spent prompting, and the tools' struggle with the implicit context of complex codebases. Developers accepted less than 44% of AI-generated code, spending roughly 9% of their time cleaning outputs to make them fit for their projects.
METR researchers Nate Rush and Joel Becker noted that while AI reduced time spent actively coding or researching, these gains were nullified by the overhead of managing the tool. They cautioned that these results are a point-in-time measurement and may not apply to junior developers or simpler projects. Some participants suggested the results might be influenced by a learning curve, as efficiencies may only surface after several hundred hours of usage. Other analysts noted that the findings highlight a need for intentional measurement and cultural investment to unlock AI's full value in enterprise environments.
