Why can published science look more certain than it really is?

Published science can look more certain than it is because what gets published depends partly on what was found. That's publication bias. Studies with a statistically significant result are about three times more likely to be published than equally well-run studies that found nothing. So if you only read the journals, you see the hits and miss the misses, and every effect looks bigger and more reliable than it really is.

Often nobody is hiding anything on purpose. A researcher gets a null result, assumes they made a mistake, loses interest, or guesses no journal will care, and the study goes into a drawer. The psychologist Robert Rosenthal named this the "file drawer problem" in 1979.

The stakes can be real. The antidepressant reboxetine was approved in several European countries and the UK in 2001, based on positive trials. In 2010, a German institute pooled the data, including results that had never been published, and concluded the drug was ineffective: the published data had overstated its benefit over placebo by up to 115 percent, and nearly three quarters of the patient data had gone unpublished. The story has a twist, though. A 2011 reanalysis found flaws in that verdict, and a 2018 study of 21 antidepressants judged reboxetine better than placebo, just weaker than the others.

The fix is to register a study before the data exist, so that a failure can't quietly disappear. Since 2004, top medical journals have refused drug trials that weren't registered from the start.