Why change net.inet.tcp.tcbhashsize in FreeBSD?

Solution 1:

It's more like computer science question. Especially if you want to dig into hash tables and big-O notations.

The answer is:
If you are handling many TCP sessions on sever you really want to look up connection's tcp parameters in O(1) time instead of O(n). FreeBSD uses chaining to resolve hash table collisions. So if there is lots of connection there will be lots of collisions and so instead of O(1) hash table lookup you'll need to do an linear chain lookup with O(n) complexity.

Parameter you mentioned - tcbhashsize is basically number of buckets in hash table.
On our servers it's set to pretty high values like 16384 and even higher. With that setting we are handling about 60,000 connections per server.

Each entry in hash table by itself currently on x86_64 uses 252 bytes (tcp_inpcb) + 688 bytes(tcpcb) of kernel memory for each entry (kmem size is 512G in amd64 since 7.2+ IIRC). It can be viewed via vmstat -z.

About structure of TCP Control block you can read FreeBSD sources: tcp_var.h or read TCP/IP Illustrated, Volume 2: The Implementation By Gary R. Wright, W. Richard Stevens