←back to thread

Reproducibility project fails to validate dozens of biomedical studies

(www.nature.com)
267 points rntn | 2 comments | 25 Apr 25 16:14 UTC | HN request time: 0.409s | source
Show context
N_A_T_E ◴[25 Apr 25 16:58 UTC] No.43795965[source]▶
Is there any path forward to fixing the current reproducibility crisis in science? Individuals can do better, but that won't solve a problem at this scale. Could we make systemic changes to how papers are validated and approved for publication in major journals?
replies(12): >>43796160 #>>43796211 #>>43796313 #>>43796358 #>>43796415 #>>43796725 #>>43796906 #>>43796908 #>>43796955 #>>43797084 #>>43797605 #>>43797627 #
_aavaa_ ◴[25 Apr 25 18:03 UTC] No.43796725[source]▶
Pre-registration is a pretty big one: essential you outline your research plan (what you’re looking for, how you will analyze the data, what bars you are setting for significance, etc.) before you do any research. You plan is reviewed and accepted (or denied), often by both funding agency and journal you want to submit to, before they know the results.

Then you perform the experiment exactly* how you said you would based on the pre-registration, and you get to publish your results whether they are positive or negative.

* Changes are allowed, but must be explicitly called out and a valid reason given.

https://en.wikipedia.org/wiki/Preregistration_(science)

replies(3): >>43798108 #>>43798882 #>>43803663 #
1. isthatafact ◴[26 Apr 25 13:56 UTC] No.43803663[source]▶
A huge benefit of that is that it would force the publication of null results, although there would still be no incentive for others to cite null result publications, and citations are unfortunately the main metric that determines the "value" of a scientist. Is there a way to make null result publications valuable?

Also, forcing pre-registration on everyone would be problematic because some types of research are not well-suited to strict planning and committee approval -- how would you quickly make adjustments to an experiment? how would you do exploratory data analysis? serendipitous discoveries would be suppressed? etc.

replies(1): >>43813929 #
2. _aavaa_ ◴[27 Apr 25 18:09 UTC] No.43813929[source]▶
You can still do exploratory data analysis, you just have to say that’s what you’re doing ahead of time.

This way, when you find one significant correlation after testing ten thousand options, you have to say you tried all ten thousand, rather than make it sounds like you only tested the one.