Tech●●●●●Difficulty 5 of 5

Why does Google budget for its own failures?

Google deliberately does not aim for services that never fail, and its engineers can tell you exactly how much failing they are allowed.

▶ Start the story

Google budgets for its own failures because chasing perfect reliability backfires. The company's own SRE book says it plainly: you might expect Google to try to build 100% reliable services, but past a certain point, increasing reliability is worse for a service and its users rather than better. Extreme reliability limits how fast new features can be developed and delivered, and it dramatically increases costs. And users often cannot tell the difference anyway: a person on a 99% reliable smartphone cannot tell 99.99% from 99.999% service reliability.

So Google's site reliability engineers set a target short of perfect, called a , or , and the gap between the target and 100% becomes a quarterly . A service with a 99.999% SLO may fail a 0.001% share of its queries in a quarter. That budget is an objective number that settles an old argument: developers are judged on how fast they ship, reliability engineers on how stable the service stays, and each has a reason to push the other. While there is error budget left, new releases can go out. If failures eat it up, releases are temporarily halted while resources go into testing and making the system more resilient.

0.001%

the failure rate a 99.999% quarterly SLO allows

The budget turns a political fight into arithmetic, and Google SRE's unofficial motto explains why that matters: "Hope is not a strategy." Its main benefit is a common incentive that lets both teams find the right balance between innovation and reliability.

Failure still happens, and when it does, Google writes a : a written record of the incident, its impact, what was done, the root causes and the follow-up actions. These are blameless. The idea is to assume that everyone involved had good intentions and did the right thing with the information they had, because if shaming prevails, people will not bring issues to light for fear of punishment.

Quiz me

0/3

  1. 1.What does an error budget decide at Google?
  2. 2.Why does Google not aim for a service that is 100% reliable?
  3. 3.Why are Google's postmortems written to be blameless?

Recap

Aim below perfect on purpose, and spend the gap as a budget: release while it lasts, fix the system when it runs out.

💡 A trick to remember it · Reliability is a bank account: ship while there is money in it, and when it is empty, repair the system rather than blame the teller.

Surprising fact · Blameless postmortems come from healthcare and avionics, where mistakes can be fatal.

Sources (3)

No source, no claim. Every fact in this lesson (15 claims) cites at least one of these.

  1. [1]Site reliability engineering · Wikipedia
  2. [2]Embracing Risk (Site Reliability Engineering, chapter 3) · Google
  3. [3]Postmortem Culture: Learning from Failure (Site Reliability Engineering, chapter 15) · Google
More lessons in 💻 Tech (3) See all tech lessons →

One more light on your map.

Get one lesson like this every day, about the things you love. Free, in two or five minutes.

Get the share card for this lesson ↗