TCP SYN Flooding Detection Method in the Linux Kernel
If I'm reading the sysctl/tcp stuff correctly, it's tripped when the number of un-ACKed syn requests exceeds the value of net.ipv4.tcp_max_syn_backlog. Specifically:
The tcp_max_syn_backlog variable tells your box how many SYN requests to keep in memory that we have yet to get the third packet in a 3-way handshake from. The tcp_max_syn_backlog variable is overridden by the tcp_syncookies variable, which needs to be turned on for this variable to have any effect. If the server suffers from overloads at peak times, you may want to increase this value a little bit.
The reason I think it's that simple is the text from tcp_syncookies:
The tcp_syncookies variable is used to send out so called syncookies to hosts when the kernels syn backlog queue for a specific socket is overflowed. This means that if our host is flooded with several SYN packets from different hosts, the syn backlog queue may overflow, and hence this function starts sending out cookies to see if the SYN packets are really legit.
To me that makes it sound like it really is as simple as the syn queue having > tcp_max_syn_backlog outstanding connections.
This article on SYN
cookies might help. You can, of course, examine the source.