←back to thread

Measuring the impact of AI on experienced open-source developer productivity

(metr.org)
688 points dheerajvs | 3 comments | 10 Jul 25 16:29 UTC | HN request time: 0.826s | source
Show context
NewsaHackO ◴[10 Jul 25 17:15 UTC] No.44523209[source]▶
So they paid developers 300 x 246 = about 73K just for developer recruitment for the study, which is not in any academic journal, or has no peer reviews? The underlying paper looks quite polished and not overtly AI generated so I don't want to say it entirely made up, but how were they even able to get funding for this?
replies(5): >>44523366 #>>44523443 #>>44523493 #>>44523508 #>>44523611 #
1. resource_waste ◴[10 Jul 25 17:50 UTC] No.44523611[source]▶
>which is not in any academic journal, or has no peer reviews?

As a philosopher who is into epistemology and ontology, I find this to be as abhorrent as religion.

'Science' doesn't matter who publishes it. Science needs to be replicated.

The psychology replication crisis is a prime example of why peer reviews and publishing in a journal matters 0.

replies(2): >>44524461 #>>44526091 #
2. bee_rider ◴[10 Jul 25 19:12 UTC] No.44524461[source]▶
> The psychology replication crisis is a prime example of why peer reviews and publishing in a journal matters 0.

Specifically, it works as an example of a specific case where peer review doesn’t help as much. Peer review checks your arguments, not your data collection process (which the reviewer can’t audit for obvious reasons). It works fine in other scenarios.

Peer review is unrelated to replication problems, except to the extent to which confused people expect peer review to fix totally unrelated replication problems.

3. raincole ◴[10 Jul 25 21:58 UTC] No.44526091[source]▶
Peer reviews are very important to filter out obviously low effort stuff.

...Or should I say "were" very important? With the help of today's GenAI every low effort stuff can look high effort without much extra effort.