What makes a superforecaster, and how did the Good Judgment Project find them?

A superforecaster is a person whose forecasts can be shown, by statistics, to have been consistently more accurate than those of the general public or of experts. The term comes out of the research of the psychologist Philip Tetlock and the Good Judgment Project, a forecasting tournament that picked its stars by their track record rather than their titles.

The story starts with a disappointment. Between 1984 and 2003, Tetlock ran small forecasting tournaments with 284 experts, including government officials, professors and journalists, whose views ranged from Marxists to free-marketeers. Across roughly 28,000 predictions, they were often only slightly more accurate than chance. Forecasters with the biggest media profiles did especially badly, which suggested an inverse relationship between fame and accuracy.

That finding caught the attention of the US intelligence community, and in 2011 its research agency, IARPA, launched a four-year tournament. The Good Judgment Project entered it with talented amateurs rather than subject experts, trained them in forecasting practice and overcoming biases, and combined their predictions with an algorithm. It won both seasons, 35 to 72 percent more accurate than any other team. Its top forecasters were reportedly about 30 percent better than analysts with access to secret data.

The headline can be overstated, and being a superforecaster does not mean always being right. They think in probabilities. The BBC noted that they gave a leave vote 23 percent in the month of the 2016 Brexit referendum, and when a 23 percent event happens, that is not automatically an error. The claim of an edge over experts has also been overstated in some sources, and critics such as Nassim Nicholas Taleb dispute whether it is useful at all.