ssh delay when connecting
Solution 1:
It's most often dns problem. Try setting 'UseDNS no' in sshd_config.
Solution 2:
It could possibly be a reverse-dns lookup delay. If your connecting host doesn't have an DNS entry, try adding an entry for your source system in /etc/hosts on the server you're connecting to.
Solution 3:
In addition to 3molos answer you could also make sure that your box has correct DNS servers setup in /etc/resolve.conf, given that you have a DNS server available on the network. This will also prevent the delay,