How do engineers build systems that keep working when parts fail?

They assume parts will fail and build so that no single failure can stop the whole. High availability is a property of systems that aim to ensure an agreed level of performance, usually uptime, for a higher than normal period. The core recipe has three parts: remove every single point of failure by adding redundancy, so that one broken component does not mean a broken system; make the switch from the broken part to its spare reliable; and detect failures as they occur.

The second part is the sneaky one. In redundant systems, the crossover point itself tends to become a single point of failure. A spare tyre is useless if the jack is missing. And the third matters because, if the first two work, a user may never see a failure, but the maintenance team must.

Then there is a famous twist: practise failing. In 2011 Netflix invented Chaos Monkey, a tool that intentionally disables computers in its own production network to test how the remaining systems respond. Netflix said that knowing failures would happen frequently created a strong alignment among engineers to build redundancy and automation to survive them.

The ambition is measured in 'nines' of availability. A rule of thumb says five nines allows about five minutes of downtime per year, and each nine fewer multiplies that by ten: four nines is 50 minutes and three nines is 500 minutes. The exact sums are close: a year has 8,760 hours, so 99.9% leaves about 8.8 hours, roughly 526 minutes, 99.99% about 53 minutes and 99.999% about 5.3 minutes. Every extra nine leaves ten times less room for failure, which is why engineers spend as much time on how things fail as on how they work.