How do high traffic sites service more than 65535 TCP connections?

If there is a limit on the number of ports one machine can have and a socket can only bind to an unused port number, how do servers experiencing extremely high amounts (more than the max port number) of requests handle this? Is it just done by making the system distributed, i.e., many servers on many machines?

Solution 1:

You misunderstand port numbers: a server listens only on one port and can have large numbers of open sockets from clients connecting to that one port.

On the TCP level the tuple (source ip, source port, destination ip, destination port) must be unique for each simultaneous connection. That means a single client cannot open more than 65535 simultaneous connections to a single server. But a server can (theoretically) serve 65535 simultaneous connections per client.

So in practice the server is only limited by how much CPU power, memory etc. it has to serve requests, not by the number of TCP connections to the server.

Solution 2:

You are mistaken - the socket's uniqueness is determined by five factors:

1. the local IP address
2. the local port number
3. the remote IP address
4. the remote port number
5. the transfer protocol (TCP/UDP)

When offering network services, 1. and 2. typically are static (e.g. IP 10.0.0.1, port 80) but unless you are expecting thousands of connections from a single client (or a single NAT gateway), you are not going to push the boundaries for the possible combinations of 3. and 4. before you run out of local resources.

So although practically a client will not use a port already in use for a connection to open a connection to a different destination IP address, port number depletion is going to be the least of your problems for nearly any application - be it on the server or client side.

The problem is a very real one with NAT gateways (routers) serving clients with a high number of open outbound connections (e.g. torrents) - there you will see port number depletion after the port pool available for NAT has been emptied. In this case the NAT gateway is unable to create any additional associations, thus effectively cutting clients off the internet.