The AI study that broke because developers wouldn't work without it

The most useful result in METR's latest productivity study is the one they could not measure. If METR had asked me this year to hand over my tasks, let it randomly pick which ones I had to do without AI, and then do those by hand, I would have quietly kept a few back. I'm not proud of that, but I would have had plenty of company.

In February, METR (the research group behind last year's much-quoted result that AI made experienced open-source developers slower) published an update on its developer productivity experiment. The headline is not a number, because they couldn't get a reliable one. The reason they couldn't is the most honest data point on AI and productivity we've had this year.

The early 2025 study found AI made tasks take 19% longer, with a confidence interval between +2% and +39%. That result travelled a long way, mostly as ammunition, skeptics calling the tools hype and fans arguing about the setup.

The follow-up was bigger, with 57 developers, 143 repositories and more than 800 tasks. Ten of those developers came back from the original study, the other 47 were new. For the returning group METR now estimates a speedup of -18%, with a confidence interval between -38% and +9%. For the new recruits the estimate is -4%, with an interval between -15% and +9%. Both intervals include zero, so neither one rules out the tools making no difference.

METR then said something researchers rarely say about their own work, that the data from the new experiment "gives us an unreliable signal of the current productivity effect of AI tools." I respect that, it would have been easy to publish the point estimates under a confident headline and let the internet do the rest.

The refusal is the result

The signal broke because between 30% and 50% of developers told METR they were "choosing not to submit some tasks", since they did not want to do them without AI. Random assignment only works if every task goes into the pool. If people hold back exactly the tasks where they most want help, the pile done by hand is the easier one, and the comparison stops measuring the tools. It measures which work developers were willing to give up AI for.

As a measurement, that's a flaw. As a finding, it's the clearest answer in the study, because developers voted with their tasks. Whatever the stopwatch says, a large share of them didn't want to go back to working without AI, even for an experiment, even for part of their work. That says more about how they value the tools than a timing number can.

It also fits the wider industry. In The Pragmatic Engineer's 2026 tooling survey, 95% of respondents use AI tools at least weekly and 55% say they regularly use AI agents. Yes, that's a survey of the newsletter's own subscribers rather than a random sample, so I wouldn't stretch it, but when nearly everyone you could recruit uses these tools every week, a control group without them is a group you have to pay to pretend.

There's a second problem, and it matters more for teams than for researchers. METR says its time measurements are unreliable for developers who use several agents at once. Some developers also told METR it was hard to report time spent with agentic tools, because they would often work on an unrelated task while waiting for the agent to finish.

That matches how the work feels now. You start an agent on a migration, review someone else's change while it runs, answer a question, then come back and correct it. A task like that can sit open for hours on the clock while taking a fraction of that in actual attention. The real cost is attention, and attention is now split across several things at once. Time per task was always a rough proxy, with parallel agents it describes almost nothing.

What I would watch instead

So let's not waste time with stopwatches. If I wanted to know whether AI was helping my own team, I'd look at what happens to the work afterwards. I think four measures hold up better: changes that survive a month, review time, rework and incidents.

A change that survives a month is code still in place four weeks later, not reverted or quietly rewritten. Review time is how long a change waits for a human and how long that human spends reading it. It matters because if agents produce more code, review is where it piles up, and that is usually where the real limit sits. Rework means the follow-up fixes to the same area in the weeks after a change ships. Incidents are production problems traced back to a change, and how long each took to resolve.

None of these are new, and that's deliberate. They told you whether software work was good before AI, and they still do. They don't care whether the code came from a person, an agent or three agents at once, only whether it held up once it shipped.

METR says it's changing its study design, with options like shorter and more intensive experiments, observational data on how developers use AI, fixed-task experiments and time-use studies. Good, I want better numbers too, but teams should not wait for a single percentage to settle the argument.

The useful question isn't whether AI makes us faster but where our attention goes now, and whether what we ship is still standing a month later. We ask the same of the apps people build on Workbricks. What matters is whether the work got done right, not how much AI was involved.