'ssh localhost' gives 'Permission denied (publickey).'
Please, check /var/log/auth.log for possible errors during connection. This line
Oct 25 09:49:47 me-myubuntu sshd[16442]: User me from localhost not allowed because not listed in AllowUsers
indicates the problem.