How to get a linux network log?

We have a java server running in linux at a specific port that accepts persistent connections for thousands and thousands of users. Recently our clients are not able to connect with a time out error. We suspect the traffic is getting too high but our java log actually shows that not many are connected in per second.

We suspect that could it be that too many are trying at the same time and they are basically dropped at the OS level and therefore the java program never really gets a chance to accept the connection? Is there some sort of log in linux that can show someone trying to hit a socket?

Solution 1:

iptables -I INPUT -p tcp --dport some_port -j LOG then
tail -f /var/log/messages
Afterwards, to see how much data has been hit by that rule: iptables -L -n -v
Or you could run tcpdump and grep out the ports.

Solution 2:

When I have really nasty network problems, I tend to fire up wireshark. For me, there's no better network diagnostic tool when I have to get down to the nitty-gritty details. And don't worry if you can't install it on either source or destination box; you can run tcpdump -w to write packet data to a file on start and/or endpoint, and feed file to wireshark on another box at your convenience.

Solution 3:

watch -n1 -d "netstat -an | grep ESTABLISHED | wc -l"

shows the number of currently established connections.

Compare this with your active ulimit settings, and of course with the maximum number of connections your java app can handle.