←back to thread

Measuring the impact of AI on experienced open-source developer productivity

(metr.org)
690 points dheerajvs | 4 comments | 10 Jul 25 16:29 UTC | HN request time: 0.214s | source
Show context
pera ◴[10 Jul 25 18:54 UTC] No.44524261[source]▶
Wow these are extremely interesting results, specially this part:

> This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%.

I wonder what could explain such large difference between estimation/experience vs reality, any ideas?

Maybe our brains are measuring mental effort and distorting our experience of time?

replies(7): >>44524872 #>>44524974 #>>44525239 #>>44525349 #>>44528508 #>>44528626 #>>44530564 #
longwave ◴[10 Jul 25 20:10 UTC] No.44524974[source]▶
I also wonder how many of the numerous AI proponents in HN comments are subject to the same effect. Unless they are truly measuring their own performance, is AI really making them more productive?
replies(1): >>44526704 #
malfist ◴[10 Jul 25 23:05 UTC] No.44526704[source]▶
How would you even measure your own performance? You can go and redo something, forgetting everything you did along the way the first time
replies(1): >>44526806 #
1. jwrallie ◴[10 Jul 25 23:21 UTC] No.44526806[source]▶
You could go the same way as the study, flip a coin to use AI or not, write down the task you just did, the time you thought the task took you and the actual clock time. Repeat and self-evaluate.
replies(1): >>44527279 #
2. malfist ◴[11 Jul 25 00:37 UTC] No.44527279[source]▶
Sample size of 16 is already hard enough to draw conclusions from. Sample size of 1 is even worse.
replies(2): >>44529007 #>>44537965 #
3. JelteF ◴[11 Jul 25 06:38 UTC] No.44529007[source]▶
It's the most representative sample size if you're interested in your own performance though. I really don't care if other people are more productive with AI, if I'm the outlier that's not then I'd want to know.
4. sarchertech ◴[11 Jul 25 23:47 UTC] No.44537965[source]▶
Sample of 16 is plenty if the effect is big enough.

It’s also not a sample size of 1, it’s a sample size of however many tasks you do because you don’t care about measuring the effect AI has on anyone but yourself if you’re trying to discern how it impacts you.