Why would armoring a bomber's most-hit spots be a mistake?

Because the only bombers you can inspect are the ones that made it home. A plane hit somewhere fatal never returns to be counted, so the holes on the survivors mostly mark places where a plane can take a hit and still fly. Armoring there would protect the wrong spots. This trap is called survivorship bias: a statistical error that comes from concentrating on things that passed a selection process over those that did not.

During World War II, the statistician Abraham Wald, a member of Columbia University's Statistical Research Group, worked on exactly this problem. The group issued eight memoranda on analyzing damage to combat aircraft, and since the data only ever came from planes that had returned, the work directly addressed survivorship bias. Wald derived a way to estimate the damage distribution of all the aircraft that flew from the damage on the ones that came back; his work is considered seminal in the young discipline of operational research.

The famous version, in which the military plans to armor the bullet-riddled fuselage and Wald tells them to armor the motors instead, is harder to pin down. The mathematician Bill Casselman says the story might well be true and has a solid germ of truth, but very little evidence for its best bits: Wald's memoranda are severely technical and say nothing about what the military should do. Wald's own path had been shaped by exclusion: he could not get a university post in Austria because of discrimination against Jews, and emigrated to the United States.

The same blind spot shows up far beyond bombers. A parapsychologist once believed he had found genuine psychic ability in a handful of subjects, while poorly performing subjects were subtly disregarded. In finance, performance studies that leave out failed companies because they no longer exist can make results look too optimistic. A famous study even seemed to show Oscar winners living almost four years longer than other actors, until a fairer method shrank the gap to about a year and erased its statistical significance.