←back to thread

504 points puttycat | 2 comments | | HN request time: 0.415s | source
Show context
MarkusQ ◴[] No.46182578[source]
This is as much a failing of "peer review" as anything. Importantly, it is an intrinsic failure, which won't go away even if LLMs were to go away completely.

Peer review doesn't catch errors.

Acting as if it does, and thus assuming the fact of publication (and where it was published) are indicators of veracity is simply unfounded. We need to go back to the food fight system where everyone publishes whatever they want, their colleagues and other adversaries try their best to shred them, and the winners are the ones that stand up to the maelstrom. It's messy, but it forces critics to put forth their arguments rather than quietly gatekeeping, passing what they approve of, suppressing what they don't.

replies(5): >>46182677 #>>46182703 #>>46182872 #>>46183274 #>>46183453 #
ulrashida ◴[] No.46182677[source]
Peer review definitely does catch errors when performed by qualified individuals. I've personally flagged papers for major revisions or rejection as a result of errors in approach or misrepresentation of source material. I have peers who say they have done similar.

I'm not sure why you think this isn't the case?

replies(1): >>46187474 #
1. MarkusQ ◴[] No.46187474[source]
Poor wording on my part.

I should have said "Peer review doesn't catch _all_ errors" or perhaps "Peer review doesn't eliminate errors".

In other words, being "peer reviewed" is nowhere close to "error free," and if (as is often the case) the rate of errors is significantly greater than the rate at which errors are caught, peer review may not even significantly improve the quality.

https://pmc.ncbi.nlm.nih.gov/articles/PMC1182327/

replies(1): >>46209503 #
2. ulrashida ◴[] No.46209503[source]
Thanks for clarifying, I fully agree with your take. Peer review helps, particularly where reviewers are equipped and provided the time to do the role correctly.

However, it is not alone a guarantor of quality. As someone proximate to academia its becoming obvious that many professors are beginning to throw in the towel or are sharply reducing their time verifying quality when faced with the rising tide of slop.

The window for avoiding the natural consequences of these trends feels like it is getting scarily small.

Thanks for taking the time to reply!