SSH server not working (respawns until stopped)

I have a running Ubuntu Server 10.04.1. When I tried to login to the server via ssh, I could not. Instead, I got connection refused error. I tried to ping the machine and I got reply! So, the clear reason is that SSH daemon is stopped.

After reboot, I was able to login to my server via ssh. After some time, I looked at my logs /var/log/syslog and found the following records:

Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2465) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2469) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2473) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2477) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2481) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2485) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2489) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2493) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2497) terminated with status 255
Jan 16 10:57:09 myserver init: ssh main process ended, respawning
Jan 16 10:57:09 myserver init: ssh main process (2501) terminated with status 255
Jan 16 10:57:09 myserver init: ssh respawning too fast, stopped

I searched for a similar problem/solution. Some people said that this is caused by the SSH daemon trying to start before networking and they suggest to change ListenAddress in /etc/ssh/sshd_config to be 0.0.0.0. I think this is not the cause in my case, because my problem occurs after system is up and running.

Any idea what is causing this? This is Ubuntu Server and it should be running and accessed remotely using SSH.

UPDATE:

Here is the log snippet I found in /var/log/auth.log.

Jan 16 10:56:38 myserver sudo:     user : TTY=pts/0 ; PWD=/home/user ; USER=root ; COMMAND=/usr/bin/vim /etc/ssh/sshd_config
Jan 16 10:57:09 myserver sudo:     user : TTY=pts/0 ; PWD=/home/user ; USER=root ; COMMAND=/etc/init.d/ssh reload
Jan 16 10:57:09 myserver sshd[1465]: Received SIGHUP; restarting.
Jan 16 10:57:09 myserver sshd[2461]: Server listening on 0.0.0.0 port 22.
Jan 16 10:57:09 myserver sshd[2465]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2465]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2469]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2469]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2473]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2473]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2477]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2477]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2481]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2481]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2485]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2485]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2489]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2489]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2493]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2493]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2497]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2497]: fatal: Cannot bind any address.
Jan 16 10:57:09 myserver sshd[2501]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Jan 16 10:57:09 myserver sshd[2501]: fatal: Cannot bind any address.

It seems that this error started to appear after I reloaded the SSH daemon. Should I avoid using ssh reload and use ssh restart instead?

I just had the same problem on my 12.04 box. I.e. same symptoms. Alas, it always happened when I introduced the ListenAddress clause with the inet and inet6 addresses in sshd_config. In short, this appears to be a symptom of a malformed sshd_config - although the log files didn't state anything like that.

Troubleshooting sshd

What I find generally very useful in any such cases is to start sshd without letting it daemonize. The problem in my case was that neither syslog nor auth.log showed anything meaningful.

When I started it from the terminal I got:

# $(which sshd) -Ddp 10222
/etc/ssh/sshd_config line 8: address family must be specified before ListenAddress.

Much better! This error message allowed me to see what's wrong and fix it. Neither of the log files contained this output.

NB: at least on Ubuntu the $(which sshd) is the best method to satisfy sshd requirement of an absolute path. Otherwise you'll get the following error: sshd re-exec requires execution with an absolute path. The -p 10222 makes sshd listen on that alternative port, overriding the configuration file - this is so that it doesn't clash with potentially running sshd instances. Make sure to choose a free port here.

This method has helped me many many times in finding issues, be it authentication issues or other types. To get really verbose output to stdout, use $(which sshd) -Ddddp 10222 (note the added dd to increase verbosity). For more debugging goodness check man sshd.

The main advantage of this method is that it allows you to check the sshd configuration without having to restart the sshd on the default port. Normally this should not interfere with existing SSH-connections, but I've seen it. So this allows one to validate the configuration file prior to - potentially - cutting off ones access to a remote server (for example I have that for some VPS and even for physical servers where I need to pay extra to get out-of-band access to the machine).

You should check to see what happened just before SSH started floundering in syslog. If the networking subsystem died, that could explain why sshd started failing.

I would also check /var/log/auth.log. It's sshd's log and it might give you a better error message.

This appears to be the result of bug #687535, which was fixed recently in natty, and has been uploaded to both maverick and lucid as a proposed update.

https://bugs.launchpad.net/ubuntu/lucid/+source/openssh/+bug/687535

I'd encourage everyone to go there, try the test case (search for TEST CASE), and post your results both before and after installing the proposed fix. That will help the SRU team decide that verification has been done and release it as an update.