More than 3 seconds "Initial Connection" time (webpagetest.org)
Solution 1:
Check reverse DNS -- the logging on the web server may be trying to lookup the name of the client, and timing out.
Test it by putting a known IP address in the /etc/hosts table, make sure you can ping by name (lookup working -- if not check the /etc/resolv.conf).
Solution 2:
This makes me think about MTU issues. Ping will work fine because it sends shorts packets, but what happens when you ping with larger packets ? And you did disable jumbo frames, did you ?
ping -s 1500 your_server
Either that, or a firewall in the middle does not like your initial TCP parameters or MTU, and/or block 'fragmentation needed' ICMP errors.