Measuring the impact of AI on experienced open-source developer productivity

I wonder if the discrepancy is that it felt like it was taking less time because they were having to do less thinking which feels like it is easier and hence faster.

Even so... I still would be really surprised if there wasn't some systematic error here skewing the results, like the developers deliberately picked "easy" tasks that they already knew how to do, so implementing them themselves was particularly fast.

Seems like they authors had about as good methodology as you can get for something like this. It's just really hard to test stuff like this. I've seen studies proving that code comments don't matter for example... are you going to stop writing comments? No.

> which feels like it is easier and hence faster.

We explore this factor in section (C.2.5) - "Trading speed for ease" - in the paper [1]. It's labeled as a factor with an unclear effect, some developers seem to think so, and others don't!

> like the developers deliberately picked "easy" tasks that they already knew how to do

We explore this factor in (C.2.2) - "Unrepresentative task distribution." I think the effect here is unclear; these are certainly real tasks, but they are sampled from the smaller end of tasks developers would work on. I think the relative effect on AI vs. human performance is not super clear...

[1] https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf

Sounds like you've thought of everything!