How did one faulty update crash 8.5 million computers?

One faulty update crashed 8.5 million computers because it reached them all at once, and it did its damage deep inside Windows. On 19 July 2024 the cybersecurity company CrowdStrike distributed a faulty update to its Falcon Sensor software. Falcon works at the operating system kernel level on each computer to detect threats, so when it failed, the whole machine crashed and could not restart properly. Microsoft estimated that 8.5 million Windows devices were affected, less than one percent of all Windows machines, but enough to be called the largest outage in the history of information technology.

The trigger was a small mismatch. CrowdStrike's own analysis says the sensor expected only 20 input values, but a new update asked it to look at a 21st. The attempt to access the 21st value produced an out-of-bounds memory read beyond the end of the input array and resulted in a system crash. The mismatch had slipped through several layers of validation and testing, in part because earlier tests used wildcard criteria for that 21st input, so it was never really read.

The reach was what made it historic. Airlines, airports, banks, hotels, hospitals, stock markets, broadcasters and government services were all affected. In India, handwritten boarding passes were issued throughout the outage. The CrowdStrike software did not give customers a way to delay installing its content files, so organisations had no chance to wait and see.

CrowdStrike reverted the update at 05:27 UTC, and devices that booted after the revert were not affected. But many computers had to be fixed manually, one by one, so outages lingered for days. CrowdStrike's fix for the future is staged deployment: new updates are canary-tested, then promoted through wider rings or rolled back if problems are detected.