What does it mean for a forecast to be well calibrated, and how is it scored?

A forecast is well calibrated when its probabilities match reality over the long run. Philip Dawid put it this way: a forecaster is well calibrated if, of the events to which they assign a probability of 30 percent, the long-run proportion that actually occurs turns out to be 30 percent. Say "70 percent" a hundred times, and about seventy of those things should happen.

The Brier score is the standard way to keep score. Proposed by Glenn W. Brier in 1950, it measures the accuracy of probabilistic predictions. It is, in effect, the mean squared error of the forecast: for each prediction you take the gap between the probability you gave and what happened (1 if it happened, 0 if not), square it, and average over all your predictions.

Squaring makes it harsh on confident mistakes. The score runs from zero to one, and lower is better. If you forecast 100 percent rain and it stays dry, you score 1, the worst possible. Forecast 70 percent and be right, and you score 0.09. A forecaster who says 50 percent every time scores 0.25 whatever happens, which makes a handy benchmark: a forecaster with real skill should beat it.

Most of us are not naturally calibrated. In research on overconfidence, people answering hard questions about unfamiliar topics were more sure than they deserved to be. In one spelling task, subjects were right about 80 percent of the time while claiming to be 100 percent certain. Keeping score is how forecasters find out which side of the line they are on.