How does a website spread millions of visitors across machines?

A website spreads visitors across machines with a load balancer: a program, or a device, that stands in front of many identical servers and hands each incoming request to one of them. Load balancing is the process of distributing tasks over a set of resources so that processing is more efficient, response times improve, and no machine is overloaded while others sit idle. One of its most common uses is to provide a single internet service from multiple servers, a server farm.

The visitor never notices. A server-side load balancer listens where clients connect, forwards each request to one of the backend servers, and passes the reply back, so the client never knows about the machines behind it. The rules can be simple. In round robin, the first request goes to the first server, the next to the second, and so on down to the last, then it starts again. Least connections instead picks the server with the fewest active connections.

There is also a way to do it with no dedicated machine at all. Round-robin DNS answers address lookups with a list of addresses for several servers that host identical services. It has a weak spot. Some resolvers re-order the list to prefer numerically closer networks, and when applied to IPv4, a bug in Windows Vista defeated round-robin load balancing altogether.

Good balancers also watch their servers. With health checking, the balancer polls servers and removes failed ones from the pool, so visitors are sent only to machines that answer. And because the balancer is itself a machine, it must not become a single point of failure: balancers are usually run in high-availability pairs.