SSH public key login fails without pattern
(previously posted at stackoverflow by error)
I'm running a bunch of servers with Ubuntu 14.04.1 (sun,hyperion,...) all of which use public keys (OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014 on all machines) for rsync without problems. Almost all...
One connection fails without any changes in the configs or keys. Then I'll try to re-add the keys, check for ECDSA, reboot/restart ssh and it works again. Or it doesn't. In this case I just wait a random amount of time (1h up to 3 months) an do the same. This time it fixes the problem - for a while.
The relevant parts of a ssh -vvv diff:
Successful connection
debug1: Host 'hyperion.internal' is known and matches the ECDSA host key.
debug1: Found key in /home/bar/.ssh/known_hosts:20
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/bar/.ssh/id_rsa (0x7f..),
debug2: key: /home/bar/.ssh/id_dsa ((nil)),
debug2: key: /home/bar/.ssh/id_ecdsa ((nil)),
debug2: key: /home/bar/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/bar/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp 95:...
debug3: sign_and_send_pubkey: RSA 95:...
debug1: key_parse_private2: missing begin marker
debug1: read PEM private key done: type RSA
debug1: Authentication succeeded (publickey).
Authenticated to hyperion.internal ([172.16.0.10]:22).
Failed connection
debug1: Host 'hyperion.internal' is known and matches the ECDSA host key.
debug1: Found key in /home/bar/.ssh/known_hosts:20
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/bar/.ssh/id_rsa (0x7f..),
debug2: key: /home/bar/.ssh/id_dsa ((nil)),
debug2: key: /home/bar/.ssh/id_ecdsa ((nil)),
debug2: key: /home/bar/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/bar/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/bar/.ssh/id_dsa
debug3: no such identity: /home/bar/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/bar/.ssh/id_ecdsa
debug3: no such identity: /home/bar/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/bar/.ssh/id_ed25519
debug3: no such identity: /home/bar/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
Things that I've checked several times:
- permissions for .ssh/ and id_rsa on all machines
- that I'm using the right keys
- that
ssh-copy-id -i /home/bar/.ssh/id_rsa [email protected]copies the right keys to the right authorized_hosts file
What didn't really help but added to the vodoo/heisenbug effect:
- rebooting the machines
- restarting the ssh service
- fiddling with global ssh options
I've pasted the full logs with some redacted info at pastebin: wall of log
Solution 1:
The problem has been resolved, it wasn't ssh-related at all:
hyperion.internal had an encrypted home, so the key-lookup failed when it was not mounted to /home/europe.
In hindsight quite obvious, but it accounts for the heisenbug effect of not failing when observing the logs on the machine (while being logged in, of course...)
Hope this helps at least some else.
Solution 2:
debug1: Offering RSA public key: /home/bar/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
This indicates the server didn't accept your private key. Unfortunately, the server doesn't provide the client with any more detail about why it didn't accept the key, so you really need to troubleshoot this on the server.
I would start by checking the syslogs in /var/log on the server for any messages from sshd indicating why it rejected the authentication attempt.
If you have root access on the remote server, you can run a debugging instance of sshd and then connect to it with a client. On the remote server, become root and run /path/to/sshd -d -p 2222. This will launch an instance of sshd which listens on port 2222. It will accept one connection, and it will print debugging information to your terminal.
Then, on the client, run ssh as normally but include -p 2222 to connect to the correct port. If the login fails, check the debugging output printed by the server.
Solution 3:
For me, this was a permissions issue involving the Home directory as well. The permissions for the Home directory on the destination server was set to 775. From what I discovered, the Home directory permissions must be set to 755 or less. This sets it to where no user other than the owner of the home directory is allowed to have write permissions.
Solution 4:
It looks like a server issue:
debug1: Offering RSA public key: /home/bar/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,password
The server does not seem to be sending a reply here. I would try cranking the debug up to 11 on the server side and see what it is whining about.
How much time elapses after the publickey packet is sent waiting for the reply when it fails? If it i