Reproducibility project fails to validate dozens of biomedical studies

This doesn’t really surprise me at all. It’s an unrelated field, but part of the reason I got completely disillusioned with research to the point I switched out of a program with a thesis was because I started noticing reproducibility problems in published work. My field is CS/CE, generally papers reference publicly available datasets and can be easily replicated… except I kept finding papers with results I couldn’t recreate. It’s possible I made mistakes (what does a college student know, after all), but usually there were other systemic problems on top of reproducibility. A secondary trait I would often notice is a complete exclusion of [easily intuited] counter-facts because they cut into the paper’s claim.

To my mind there is a nasty pressure that exists for some professions/careers, where publishing becomes essential. Because it’s essential, standards are relaxed and barriers lowered, leading to the lower quality work being published. Publishing isn’t done in response to genuine discovery or innovation, it’s done because boxes need to be checked. Publishers won’t change because they benefit from this system, authors won’t change because they’re bound to the system.

The state of CS papers is truly awful, as they're uniquely poised to be 100% reproducible. And yet my experience aligns with yours in that they very rarely are.

I can second this, even availability of the code is still a problem. However, I would not say CS results are rarely reproducible, at least from the few experineces I had so far, but I heard of problematic cases from others. I guess it also differs between fields.

I want to note there is hope. Contrary to what the root comment says, some publishers try to endorse reproducible results. See for example the ACM reproducibility initiative [1]. I have participated in this before and believe it is a really good initiative. Reproducing results can be very labor intensive though, loading a review system already struggling under massive floods of papers. And it is also not perfect, most of the time it is only ensured that the author-supplied code produces the presented results, but I still think more such initiatives are healthy. When you really want to ensure the rigor of a presented method, you have to replicate it, i.e., using a different programming language or so, which is really its own research endeavor. And there is also a place to publish such results in CS already [2]! (although I haven‘t tried this one). I imagine this may be especially interesting for PhD students just starting out in a new field, as it gives them the opportunity to learn while satisfying the expectation of producing papers.

[1] https://www.acm.org/publications/policies/artifact-review-an... [2] https://rescience.github.io

Even more ridiculous is the number of papers that do not include code. Sure, maybe Google cannot offer an environment to replicate the underlying 1PB dataset, but for mortals, this is rarely a concern.

Even better is when the paper says code will be released after publication, but they cannot be bothered to post it anywhere.

All it takes is 14 grad students studying the same thing targeting a 95% confidence interval for, on average, one to stumble upon a 5% case. Factor in publication bias and you get a bunch of junk data.

I think I heard this idea from Freakonomics, but a fix is to propose research to a journal before conducting it and being committed to publication regardless of outcome.

A great idea. Also known as a pre registered study.

https://en.m.wikipedia.org/wiki/Preregistration_(science)

Not familiar with this idea, but this idea is commonly applied for grant applications: only apply for a grant when you finished the thing you promise to work on. Then use the grant money to prototype the next five ideas (of which maybe one works), because science is about exploration.

This same post appears at the top of every single HN story on reproducibility. “I was a student in [totally unrelated field] and found reproducibility to be difficult. I didn’t investigate it deeply and ultimately I left the field, not because I was unsuccessful, of course, but because I understood deeply despite my own extremely limited experience in the area that all of the science was deeply flawed if not false.”

Imagine the guy who got a FAANG job and made it nine weeks in before washing out, informing you how the entire industry doesn’t know how to write code. Maybe they’re right and the industry doesn’t know how to write code! But I want to hear it from the person who actually made a career, not the intern who made it through part of a summer.

This seems like a straw-man. The stories are much more complex than this (in my experience/opinion), usually directly reporting about immoral acts by peers, lack of support, unfair/inequal treatment, hypocrisy, and so on. The event of the failed reproduction is at best an intermezzo.

Not to mention that we know a lot of overhyped results did fail replication and then powerful figures in academia did their best to pretend that still their thrones were not placed on top of sandcastles.

Most pharma / medicine studies are pre-registered now. Sometimes the endpoints change based on what the scientists are seeing, but if they're worth their salt, they still report the original scoped findings as well.

> A secondary trait I would often notice is a complete exclusion of [easily intuited] counter-facts because they cut into the paper’s claim.

It's lack of industry experience. I complained about this is a recent comment here: https://news.ycombinator.com/item?id=43769856

Basically, in SE anyway, the largest number of publications are authored by new graduates.

Think about how clueless the new MSc or PhD graduate is when they join your team: thesebare the majority of authors.

The system is set up to incentivise the wrong thing.

The problem is the negative feedback cycle: someone who has spent decades in academia and is highly published, almost by definition alone, has not experienced the pains of industry practitioners.

Their findings are often irrelevant to industry at best and contradictory at worst.

Of course I'm talking almost solely about SE.

>but a fix is to propose research to a journal before conducting it and being committed to publication regardless of outcome.

Does not fix the underlying issue. Having a "this does not work" paper on your resume will do little for your career. So the incentives to make data fit a positive hypothesis are still there.

That is categorically not true. Showing why something does not work (or is not advantageous over other methods) demonstrates you know how to properly conduct research which is good for ones resume.

The paper is irrelevant and will never get cited. There is essentially zero benefit to your career as it is nothing more than a single bullet point on your resume.

Discovering something that works is significant, discovering something that does not work is irrelevant.

Can you name a single scientist, e.g. from your field, who is known for showing that something does not work?

I would encourage you to learn more about how science and research is conducted before continuing to make uninformed comments.