How can a poll of a thousand people speak for millions, when one of 2.4 million got it wrong?

In 1936 the Literary Digest magazine mailed ten million questionnaires, and 2.38 million came back. Before then its presidential poll, run since 1916, had always picked the winner. This time it predicted that Republican Alf Landon would win, with 57.08% of the vote and 370 electoral votes. In November, Roosevelt won re-election in a landslide, taking every state except Maine and Vermont. The magazine's error was 39.08 points in the popular vote margin, and it folded within 18 months.

The size of the poll was never the problem. The Digest had surveyed its own readers first, then registered automobile owners and telephone users, groups with incomes well above the national average. And later research found that non-response bias was the primary contributor to the size of the error: people who strongly disliked Roosevelt were more likely to answer. Meanwhile George Gallup, an advertising executive who had begun a scientific poll, predicted that Roosevelt would win, using a quota sample of just 50,000 people.

Sample size still matters, but in a particular way. Polls rely on the law of large numbers, and what counts is the absolute size of the sample, not the percentage of the whole population it represents. For a random sample of 1,000 people reporting a proportion around 50%, the sampling margin of error is approximately plus or minus 3%. That figure means that if the same procedure is repeated a large number of times, 95% of the time the true population average will be within 3% of the sample estimate. So the margin of error depends on how many people are asked, not on how many millions they stand for, and a poll of millions chosen badly can still be wrong by a mile.