Publishers trial paying peer reviewers – what did they find?

There's this guy I usually have on in the background on youtube who replicates chemistry experiments -- or attempts to. It's pretty rare to see him find a paper that doesn't exaggerate yields or go into enough details, and he has to guess things.

You don't exaggerate yields, you just publish the best one you get out of a dozen attempts. Chemistry is messy.

Was it perhaps "that chemist"? He has some decent videos on complete bogus papers but I don't think he does reproductions, I'd be interested in that channel if you happen to find it in your watch history.

I did a lot of chemistry for a year when I worked as a QA for a pharmaceuticals company before going to uni.

So much so that when I did Chemistry at uni I got asked if I was cheating a few times in labs, until I explained.

It's actually really hard to get any experiment perfect the first time.

Even with a year's practice of measuring and mixing and titration and all the other skills you need, I'd still get low yields, or bad results occasionally. Better than everyone else, but still not perfect.

I also noticed that the more you do a particular process, the better results you will get. Just like practicing a solo on an instrument lots, or a particular pool shot, or cooking a particular meal. There's a level of learning and experience needed for each process, not all chemistry in general.

That, in science, is called "lying".

Either you publish the range of results, the average plus standard deviation or average plus standard deviation of a subset with the exclusion criteria and exclusion range. Picking a result is a lie, plain and simple, and messiness is not an excuse.

See my comment too, you jump to lying, but as the GP said, chemistry is messy.

Any other science is messy as well.

Truck passing by on the nearby road? Oops, my physics experiment got shaken, results look messy. Lab animal caught a cold? Oops, genetics experiment now has messy data. Atmosphere is turbulent and some shitty starlink satellite passed by at the wrong moment? Oops, my stellar spectra are messy now. Imperfection in my test ingot? Oops, now my tensile strength measurements have messy data because a few ripped too early...

It is the nature of experimental science to deal with messiness. And dealing with it means being honest about it. You write it like it happened, find the problems in the messy parts of your data, exclude that and explain the why and how. Hand-picked results and just omitting data you find inconvenient is not science, its fraud.

When I am allowed to just pick one result I can show you a perpetuum mobile, cold fusion, superhuman intelligence in mice and tons of other newsworthy things...

Hence the crisis we have in science today.

As an aside, I'm working at a QC chem lab now, with results that have a direct impact on revenue calculations for clients. Therefore the reports go to accountants, therefore error bars dont't exist. We recently had a case where we reported 41.7 when the client expected 42.0 on a method that's +/- 1.5... They insisted we remeasure because our result was "impossible" The repeat gave 42.1, and the client was happy to be charged twice

Compare the yields in a typical JACS (or any high end journal) paper versus those in OrgSyn and I think it's pretty clear that yields in many papers are more than exaggerated. It's a single untraceable number and the outcome of your PhD depends on it - the incentive is very clear. Leave a bit of DCM in, weigh, high vac to get rid of the singlet at 5.30ppm and no one's any the wiser...

nileblue/red typically pulls his processes from papers that have some dubious documentation, and his results have variance with the papers'.

he's not going out of his way to reproduce papers, its just on the way of turning peanut butter into toothpaste, or something of the sorr

Can I ask if you've done any actual commercial work in any science?

From the way you're talking, I'm going to guess you're an armchair commentator.

One person performing an unfamiliar experiment once is going to get lower yields and occasional failures.

I've done scientific work in science. I've been paid for it, but by a public university, so not "commercial" in the strictest sense of the word.

Do you mean to suggest that "commercial work" in science takes shortcuts and ignores the essentials of the scientific method? Do you mean to suggest that commercial science or at least commercial chemistry writing science-like papers are all misrepresenting their results systematically? Do you think the standards for good scientific conduct do not apply to chemists or commercially working scientists? Because any of that would mean that "commercial work" in science is just fraud dressed up as science.

And yes of course an experienced experimenter will get better, easier, more consistent results, everyone knows that. The issue is not about that at all. The issue is about suppressing results and data that you don't like. Those maybe result from initial inexperience or bad luck, normal variations in measurements or whatever. You present all your data, with statistics, with an explanation, and if that explanation is "well, the initial 20 values are excluded from the reported average because of me being heavy-handed with the frobnicator" then that is fine. People can check your values, your reasoning and convince themselves that your reporting is right and your experiment works to the extend you reported. If you just say "the yield is 89%" without mentioning that all the other yields were worse, without mentioning any kind of variance, range, exclusions, you are lying. Those 89% were your single best yield, since they were best you were never able to reproduce that, so it might as well have been leftover product from improperly cleaning your glassware...

Are you really trying to convince me that all chemists are crooked like that? Or all commercial work in science is crooked?

Yep, that's the guy!