How do researchers test whether psychotherapy works, and why is it harder than testing a pill?

Testing whether psychotherapy works is harder than testing a pill, because there is no obvious fake therapy. Researchers still use the standard tool of medicine, the randomized controlled trial: people are assigned by chance to a treatment or a comparison group, which reduces selection bias and balances known and unknown factors between the groups. But in therapy trials the comparison group is often people on a waiting list, or people given regular non-specific contact or support. Researchers have to consider how to match the inert tablets and sham treatments of drug trials, and there is considerable controversy about whether trials, or more individualised methods, are the best way to evaluate therapy at all.

One trial is rarely enough, so results are pooled in a meta-analysis, which combines effect sizes across independent studies to improve statistical power. The first model meta-analysis, published in 1978 by Mary Lee Smith and Gene Glass, was about the effectiveness of psychotherapy. Hans Eysenck, whose 1952 study had found that psychotherapy generally did not seem to improve patients' outcomes, called meta-analysis an "exercise in mega-silliness". Researchers kept trying to disprove his 1952 finding, and a number of studies after 1975 presented more evidence that psychotherapy works in general. Eysenck's own record has also been questioned: fourteen of his papers were retracted in 2020, and Wikipedia reports that he fabricated data suggesting behaviour therapy could achieve results impossible to believe.

The unit of comparison is the effect size. For standardised mean differences, 0.2 to 0.5 is conventionally called small, 0.5 to 0.8 medium and above 0.8 large, though Jacob Cohen, who suggested these labels, cautioned that they are relative to the field. A 2022 umbrella review of 102 meta-analyses found that most effect sizes for both psychotherapies and medications were small, and it concluded that a "paradigm shift in research" was needed.