←back to thread

Measuring the impact of AI on experienced open-source developer productivity

(metr.org)
688 points dheerajvs | 1 comments | 10 Jul 25 16:29 UTC | HN request time: 0s | source
Show context
kokanee ◴[10 Jul 25 16:53 UTC] No.44523013[source]▶
> developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%.

I feel like there are two challenges causing this. One is that it's difficult to get good data on how long the same person in the same context would have taken to do a task without AI vs with. The other is that it's tempting to time an AI with metrics like how long until the PR was opened or merged. But the AI workflow fundamentally shifts engineering hours so that a greater percentage of time is spent on refactoring, testing, and resolving issues later in the process, including after the code was initially approved and merged. I can see how it's easy for a developer to report that AI completed a task quickly because the PR was opened quickly, discounting the amount of future work that the PR created.

replies(4): >>44523132 #>>44523767 #>>44523857 #>>44524518 #
1. qsort ◴[10 Jul 25 17:08 UTC] No.44523132[source]▶
It's really hard to attribute productivity gains/losses to specific technologies or practices, I'm very wary of self-reported anecdotes in any direction precisely because it's so easy to fool ourselves.

I'm not making any claim in either direction, the authors themselves recognize the study's limitations, I'm just trying to say that everyone should have far greater error bars. This technology is the weirdest shit I've seen in my lifetime, making deductions about productivity from anecdotes and dubious benchmarks is basically reading tea leaves.