"Possible SYN flooding" in log despite low number of SYN_RECV connections

Solution 1:

So, this is a neat question.

Initially, I was surprised that you saw any connections in SYN_RECV state with SYN cookies enabled. The beauty of SYN cookies is that you can statelessly participate in the in TCP 3-way handshake as a server using cryptography, so I would expect the server not to represent half-open connections at all because that would be the very same state that isn't being kept.

In fact, a quick peek at the source (tcp_ipv4.c) shows interesting information about how the kernel implements SYN cookies. Essentially, despite turning them on, the kernel behaves as it would normally until its queue of pending connections is full. This explains your existing list of connections in SYN_RECV state.

Only when the queue of pending connections is full, AND another SYN packet (connection attempt) is received, AND it has been more than a minute since the last warning message, does the kernel send the warning message you have seen ("sending cookies"). SYN cookies are sent even when the warning message isn't; the warning message is just to give you a heads up that the issue hasn't gone away.

Put another way, if you turn off SYN cookies, the message will go away. That is only going to work out for you if you are no longer being SYN flooded.

To address some of the other things you've done:

• net.ipv4.tcp_synack_retries:
  • Increasing this won't have any positive effect for those incoming connections that are spoofed, nor for any that receive a SYN cookie instead of server-side state (no retries for them either).
  • For incoming spoofed connections, increasing this increases the number of packets you send to a fake address, and possibly the amount of time that that spoofed address stays in your connection table (this could be a significant negative effect).
  • Under normal load / number of incoming connections, the higher this is, the more likely you are to quickly / successfully complete connections over links that drop packets. There are diminishing returns for increasing this.
• net.ipv4.tcp_syn_retries: Changing this cannot have any effect on inbound connections (it only affects outbound connections)

The other variables you mention I haven't researched, but I suspect the answers to your question are pretty much right here.

If you aren't being SYN flooded and the machine is responsive to non-HTTP connections (e.g. SSH) I think there is probably a network problem, and you should have a network engineer help you look at it. If the machine is generally unresponsive even when you aren't being SYN flooded, it sounds like a serious load problem if it affects the creation of TCP connections (pretty low level and resource non-intensive)

Solution 2:

I've faced into exactly the same problem on a fresh install of Ubuntu Oneiric 11.10 running a webserver (apache2) with a heavy loaded website. On Ubuntu Oneiric 11.10 syncookies were enabled by default.

I had the same kernel messages stating a possible SYN flood attack on the webserver port:

kernel: [739408.882650] TCP: Possible SYN flooding on port 80. Sending cookies.

At the same time, i was pretty sure, that there was no attack happening. I had this messages returning at 5min interval. This seemed just like a load peek, because an attacker would keep the load high all the time, while trying to get the server stop responding to requests.

Tuning the net.ipv4.tcp_max_syn_backlog parameter did not lead to any improvement - the messages continued at the same rate. the fact that the number of SYN_RECV connections was always really low (in my case under 250) was an indicator, that there must be some other parameter, that is responsible for this message.

I have found this bug-message https://bugzilla.redhat.com/show_bug.cgi?id=734991 on the red hat site stating that the kernel message could be as a result of a bug (or misconfiguration) on the application side. Of course the log message is very misleading! As this is not the kernel parameter that is responsible in that case, but the parameter of your application, beeing passed to the kernel.

So we should also take a look at the configuration parameters of our webserver application. Grab apache docs and go to http://httpd.apache.org/docs/2.0/mod/mpm_common.html#listenbacklog

The default value of ListenBacklog parameter is 511. (This corresponds with the number of connections, that you have observed on your red hat server. Your another server may possibly have a lower number configured.)

Apache has an own configuration parameter for the backlog queue for incoming connections. if you have a lot of incoming connections, and at any moment (just as a random thing) they arrive all together at nearly the same time, such that the webserver is not able to serve them fast enough in an appropriate way, your backlog will be full with 511 connections and kernel will fire the above message stating a possible SYN flood attack.

To solve this, i add the following line to /etc/apache2/ports.conf or one of the other .conf files, that will be loaded by apache (/etc/apache2/apache2.conf should be also ok):

ListenBackLog 5000

you should also set the net.ipv4.tcp_max_syn_backlog to a reasonable value. in my understanding, the kernel maximum will limit the value, that you will be able to configure in the apache configuration. so run:

sudo sysctl -w net.ipv4.tcp_max_syn_backlog=5000

After tuning the config, do not forget to restart your apache:

sudo service apache2 restart ( or sudo /etc/init.d/apache2 restart )

In my case, this configuration change immediately stopped the kernel warnings. I'm able to reproduce the messages by setting a low ListenBackLog value in the apache config.

Solution 3:

After some tests with kernel 3.4.9 the number of SYN_RECV connections in netstat depends on

• /proc/sys/net/core/somaxconn rounded up to the next power of 2 (e.g. 128 -> 256)
• 75% of /proc/sys/net/ipv4/tcp_max_syn_backlog if /proc/sys/net/ipv4/tcp_syncookies is set to 0 or 100% if /proc/sys/net/ipv4/tcp_syncookies is set to 1
• ListenBackLog in the apache config rounded up to the next power of 2 (e.g. 128 -> 256)

the minimum of each of this parameters is used. After changing somaxconn or ListenBackLog apache has to be restarted.

And after increasing tcp_max_syn_backlog apache has also to be restarted.

Without tcp_syncookies apache is blocking, why in this case only 75% of tcp_max_syn_backlog is the limit is strange. and increasing this paramter increases the SYN_RECV connections to 100% of the old value without restarting apache.