What causes “SYN to LISTEN sockets dropped”?
These counters ultimately come from the kernel and map to the LINUX_MIB_LISTENOVERFLOWS
and LINUX_MIB_LISTENDROPS
counters. You can see from the source of net/ipv4/tcp_ipv4.c(tcp_v4_syn_recv_sock) around line #1392 that when LINUX_MIB_LISTENOVERFLOWS
is incremented, LINUX_MIB_LISTENDROPS
will also be incremented but there are exit conditions where only the latter can be incremented so it's not a bug that they don't match.
In the same file you can see there's this code:
1291 int tcp_v4_conn_request(struct sock *sk, struct sk_buff *skb)
1292 {
1293 /* Never answer to SYNs send to broadcast or multicast */
1294 if (skb_rtable(skb)->rt_flags & (RTCF_BROADCAST | RTCF_MULTICAST))
1295 goto drop;
1296
1297 return tcp_conn_request(&tcp_request_sock_ops,
1298 &tcp_request_sock_ipv4_ops, sk, skb);
1299
1300 drop:
1301 NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_LISTENDROPS);
1302 return 0;
1303 }
So you can see at least one cause is a SYN to a broadcast or multicast address.
Usually wmem and rmem defaults are 212992 bytes. Apparently not enough on busy server. Raised to 8MB and the problem disappeared.
sysctl -w net.core.wmem_default=8388608
sysctl -w net.core.rmem_default=8388608