Web server is slow on port 80 but fast on port 81

How busy is Apache and how what is your MaxClients set at? If you are receiving a lot of incoming requests and have a low MaxClients then some requests will be delayed until an Apache child thread eventually becomes available to serve it. You can enable mod_status to get a live view of Apache requests and what each child is doing. It should tell you pretty quickly if all children are busy or not.

If it is a specific port 80 issue you can check iptables --list to see if there are any port 80 related rules that might be coming into play. It is also possible the issue is with something between your client and server (or on your client itself). Try performing some benchmarks locally on the server to see if there's any similar performance issues.

This is almost certainly a network issue and not a problem with your Apache configuration.

Port 80 traffic is probably being filtered or queued somewhere between you and the target system. Perhaps there is a proxy or other caching server in action. Port 81 traffic would naturally bypass any restrictions or special conditions placed on "normal" web traffic.

This was a SYN flood attack. The way to check this is by issuing

ss -t -a |grep SYN-RECV

I resolved this by enabling SYN cookies