dmesg SYN flood on 80 sysctl -p

Check if the packets have a distinguishing feature, for instance, all being the same size.

Usually with scripted SYN floods, they send out a "bare" packet with just the header and no payload. It ends up being a 40-byte packet (if I remember right).

If that's the case, you can simply strip all those out with iptables, since no "normal" packets look like that.

Oh and get ready for a flood of useless advice from people who don't really understand how SYN-floods actually work.

You just need to enable syncookies, and I've seen that you've already did it:

sysctl -w net.ipv4.tcp_syncookies=1

Then you can tune your OS TCP/IP stack to free system resources quicker on unused/closed sockets.

My settings:

# tunning tcp stack
sysctl -w net.ipv4.tcp_fin_timeout=30
sysctl -w net.ipv4.tcp_keepalive_time=1800
sysctl -w net.ipv4.tcp_window_scaling=0
sysctl -w net.ipv4.tcp_sack=0
sysctl -w net.ipv4.tcp_timestamps=0

sysctl -w net.ipv4.ip_conntrack_max=524288
sysctl -w net.ipv4.tcp_syncookies=1

sysctl -w net.ipv4.icmp_echo_ignore_broadcasts=1

# buffering
sysctl -w net.core.wmem_default=229376
sysctl -w net.core.wmem_max=229376

You can tune your Apache too, especially turn off KeepAlive and set a lower Timeout value:

Timeout 5
KeepAlive Off

When dealing with a lots of connections, it helps to use a web server la Nginx, Lighttpd, ... they start one single process and they allocate just a small amount of memory for each connection, Apache it's allocating one process to each connection.