Be sceptical of no registered reports

Tom Chivers at Unherd reports:

There’s a fascinating and slightly unnerving new study out in preprint, by scientists at the Eindhoven University of Technology.

Here’s the technical version. There’s a system called “Registered Reports”, in which scientists preregister their hypotheses before carrying out a study, and scientific journals agree to publish the study on the strength of the methods, rather than the results. The new study found that Registered Reports are only about 50% as likely as standard, non-RR research to confirm their hypothesis.

And here’s why it matters. At the moment, science has some profound problems. Journals tend to only publish “novel”, “exciting” results. That means that if you do an experiment to see if wine gums cause halitosis, and it comes back negative, it probably won’t get published. As I said recently, that means that journals fill up with “positive” studies and “negative” ones sit in file drawers, so the scientific literature is skewed.

The skewing is a real issue and there is value in know a hypothesis wasn’t proved.