Scaling app servers to nearly unlimited size is easy to explain but really hard in practice. It basically amounts to this:
- Balance requests using DNS anycast so you can spread load before it hits your servers
- Setup “Head End” machines with as large pipes as possible (40Gbps?) and load balance at the lowest layer you can. Balance at IP level using IPVS and direct server return. A single reasonable machine can handle a 40Gbps pipe….
- Setup a ton of HTTP-proxy type load balancers. This includes Nginx, Varnish, Haproxy etc… One of these machines can probably handle 1-5 Gbps of traffic so expect 20 or so behind each layer 3 balancer…
- Now for your app servers. Depending on if you’re using a dog slow language or not, you’ll want between 3 and 300 app servers behind each HTTP proxy.
Information source: https://news.ycombinator.com/item?id=13357278