←back to thread

Measuring the impact of AI on experienced open-source developer productivity

(metr.org)
688 points dheerajvs | 2 comments | 10 Jul 25 16:29 UTC | HN request time: 0s | source
Show context
AvAn12 ◴[10 Jul 25 18:50 UTC] No.44524235[source]▶
N = 16 developers. Is this enough to draw any meaningful conclusions?
replies(2): >>44524365 #>>44527937 #
sarchertech ◴[10 Jul 25 19:04 UTC] No.44524365[source]▶
That depends on the size of the effect you’re trying to measure. If cursor provides a 5x, 10x, or 100x productivity boost as many people are claiming, you’d expect to see that in a sample size of 16 unless there’s something seriously wrong with your sample selection.

If you are looking for a 0.1% increase in productivity, then 16 is too small.

replies(2): >>44524778 #>>44525319 #
1. biophysboy ◴[10 Jul 25 19:52 UTC] No.44524778[source]▶
Well it depends on the variance of the random variable itself. You're right that with big, obvious effects, a larger n is less "necessary". I could see individuals having very different "productivities", especially when the idea is flattened down to completion time.
replies(1): >>44533490 #
2. sarchertech ◴[11 Jul 25 15:45 UTC] No.44533490[source]▶
Individuals do have very different productivities, but they are measuring the productivity difference across a single individual.