Can't log in via SSH to any accounts using /bin/bash shell on Synology NAS

I'm trying to install bash as the default shell on a ARM Linux running on an embedded device (Synology DS212+ NAS). But there's something really wrong, and I can't figure out what it is.

Symptoms:

1) Root has /bin/bash as default shell, and can log in normally via SSH:

$ grep root /etc/passwd
root:x:0:0:root:/root:/bin/bash

$ ssh root@NAS
root@NAS's password:
Last login: Sun Dec 16 14:06:56 2012 from desktop
# 

2) joeuser has /bin/bash as default shell, and receives "Permission denied" when trying to log in via SSH:

$ grep joeuser /etc/passwd
joeuser:x:1029:100:Joe User:/home/joeuser:/bin/bash

$ ssh joeuser@localhost
joeuser@NAS's password:
Last login: Sun Dec 16 14:07:22 2012 from desktop
Permission denied, please try again.
Connection to localhost closed.

3) changing joeuser's shell back to /bin/sh:

$ grep joeuser /etc/passwd
joeuser:x:1029:100:Joe User:/home/joeuser:/bin/sh

$ ssh joeuser@localhost
Last login: Sun Dec 16 15:50:52 2012 from localhost
$ 

To make things even more strange, I can log in as joeuser using /bin/bash using the serial console (!). Also a su - joeuser as root works fine, so the bash binary itself is working fine.

In an act of despair, I changed joeuser's uid to 0 on /etc/passwd, but also didn't work, so it doesn't seem to be something permission related.

Seems that bash is doing some extra checking that sshd didn't like, and blocking the connections for non-root users. Maybe some sort of sanity checking - or terminal emulation - that is triggering the SIGCHLD, but only when called via ssh.

I already went through every single item on sshd_config, and also put SSHD in debug mode, but didn't find anything strange. Here's my /etc/ssh/sshd_config:

LogLevel DEBUG
LoginGraceTime 2m
PermitRootLogin yes
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile      %h/.ssh/authorized_keys
ChallengeResponseAuthentication no
UsePAM yes
AllowTcpForwarding no
ChrootDirectory none
Subsystem       sftp    internal-sftp -f DAEMON -u 000

And here's the output from /usr/syno/sbin/sshd -d, showing the failed attempt of joeuser trying to log in, with /bin/bash as the shell:

debug1: Config token is loglevel
debug1: Config token is logingracetime
debug1: Config token is permitrootlogin
debug1: Config token is rsaauthentication
debug1: Config token is pubkeyauthentication
debug1: Config token is authorizedkeysfile
debug1: Config token is challengeresponseauthentication
debug1: Config token is usepam
debug1: Config token is allowtcpforwarding
debug1: Config token is chrootdirectory
debug1: Config token is subsystem
debug1: HPN Buffer Size: 87380
debug1: sshd version OpenSSH_5.8p1-hpn13v11
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: read PEM private key done: type ECDSA
debug1: private host key: #2 type 3 ECDSA
debug1: rexec_argv[0]='/usr/syno/sbin/sshd'
debug1: rexec_argv[1]='-d'
Set /proc/self/oom_adj from 0 to -17
debug1: Bind to port 22 on ::.
debug1: Server TCP RWIN socket size: 87380
debug1: HPN Buffer Size: 87380
Server listening on :: port 22.
debug1: Bind to port 22 on 0.0.0.0.
debug1: Server TCP RWIN socket size: 87380
debug1: HPN Buffer Size: 87380
Server listening on 0.0.0.0 port 22.

debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 6 out 6 newsock 6 pipe -1 sock 9
debug1: inetd sockets after dupping: 4, 4
Connection from 127.0.0.1 port 52212
debug1: HPN Disabled: 0, HPN Buffer Size: 87380
debug1: Client protocol version 2.0; client software version OpenSSH_5.8p1-hpn13v11
SSH: Server;Ltype: Version;Remote: 127.0.0.1-52212;Protocol: 2.0;Client: OpenSSH_5.8p1-hpn13v11
debug1: match: OpenSSH_5.8p1-hpn13v11 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1-hpn13v11
debug1: permanently_set_uid: 1024/100
debug1: MYFLAG IS 1
debug1: list_hostkey_types: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: AUTH STATE IS 0
debug1: REQUESTED ENC.NAME is 'aes128-ctr'
debug1: kex: client->server aes128-ctr hmac-md5 none
SSH: Server;Ltype: Kex;Remote: 127.0.0.1-52212;Enc: aes128-ctr;MAC: hmac-md5;Comp: none
debug1: REQUESTED ENC.NAME is 'aes128-ctr'
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: expecting SSH2_MSG_KEX_ECDH_INIT
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug1: userauth-request for user joeuser service ssh-connection method none
SSH: Server;Ltype: Authname;Remote: 127.0.0.1-52212;Name: joeuser
debug1: attempt 0 failures 0
debug1: Config token is loglevel
debug1: Config token is logingracetime
debug1: Config token is permitrootlogin
debug1: Config token is rsaauthentication
debug1: Config token is pubkeyauthentication
debug1: Config token is authorizedkeysfile
debug1: Config token is challengeresponseauthentication
debug1: Config token is usepam
debug1: Config token is allowtcpforwarding
debug1: Config token is chrootdirectory
debug1: Config token is subsystem
debug1: PAM: initializing for "joeuser"
debug1: PAM: setting PAM_RHOST to "localhost"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user joeuser service ssh-connection method password
debug1: attempt 1 failures 0
debug1: do_pam_account: called
Accepted password for joeuser from 127.0.0.1 port 52212 ssh2
debug1: monitor_child_preauth: joeuser has been authenticated by privileged process
debug1: PAM: establishing credentials
User child is on pid 9129
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 65536 max 16384
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype [email protected] want_reply 0
debug1: server_input_channel_req: channel 0 request pty-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req pty-req
debug1: Allocating pty.
debug1: session_new: session 0
debug1: session_pty_req: session 0 alloc /dev/pts/1
debug1: server_input_channel_req: channel 0 request shell reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req shell
debug1: Setting controlling tty using TIOCSCTTY.

debug1: Received SIGCHLD.
debug1: session_by_pid: pid 9130
debug1: session_exit_message: session 0 channel 0 pid 9130
debug1: session_exit_message: release channel 0
debug1: session_by_tty: session 0 tty /dev/pts/1
debug1: session_pty_cleanup: session 0 release /dev/pts/1
Received disconnect from 127.0.0.1: 11: disconnected by user
debug1: do_cleanup
debug1: do_cleanup
debug1: PAM: cleanup
debug1: PAM: closing session
debug1: PAM: deleting credentials

Here you have the full output of sshd -dd, together with ssh -vv.

Bash:

# bash --version
GNU bash, version 3.2.49(1)-release (arm-none-linux-gnueabi)
Copyright (C) 2007 Free Software Foundation, Inc.

The bash binary was cross compiled from source. I also tried using a pre-compiled binary from the Optware distribution, but had the exact same problem. I checked for missing shared libraries using objdump -x, but they're all there.

Any ideas what could be causing this "Permission denied, please try again."? I'm almost diving in the bash source code to investigate, but trying to avoid hours chasing something that may be silly.

EDIT: adding more information about bash and the system

$ ls -la /bin/bash
-rwxr-xr-x    1 root     root        724676 Dec 15 23:57 /bin/bash

$  file /bin/bash
/bin/bash: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.14, stripped

$  uname -a
Linux NAS 2.6.32.12 #2661 Mon Nov 12 23:10:15 CST 2012 armv5tel GNU/Linux synology_88f6282_212+

$ grep bash /etc/shells
/bin/bash
/bin/bash2

Solution 1:

For future reference: after way too many hours researching and debugging this issue, I finally discovered the root cause.

The OpenSSH version used by Synology is a highly customized version, that does not behave like the original code. It has lots of hacks and ad-hoc customizations - e.g., additional checking before accepting a login to see if the SSH service is enabled within the web interface, or stripping special chars (;, |, ') from rsync commands, or... wait for it... avoiding regular users to use a shell different than /bin/sh or /bin/ash. Yeah, hard coded within the binary.

Here's the piece of code from OpenSSH 5.8p1, as distributed by Synology on their source code (DSM4.1 - branch 2636), file session.c:

void do_child(Session *s, const char *command)
{
...

#ifdef MY_ABC_HERE
   char szValue[8];
   int RunSSH = 0;
   SSH_CMD SSHCmd = REQ_UNKNOWN;

   if (1 == GetKeyValue("/etc/synoinfo.conf", "runssh", szValue, sizeof(szValue))) {
           if (strcasecmp(szValue, "yes") == 0) {
                   RunSSH = 1;
           }
   }

   if (IsSFTPReq(command)){
           SSHCmd = REQ_SFTP;
   } else if (IsRsyncReq(command)){
           SSHCmd = REQ_RSYNC;
   } else if (IsTimebkpRequest(command)){
           SSHCmd = REQ_TIMEBKP;
   } else if (RunSSH && IsAllowShell(pw)){
           SSHCmd = REQ_SHELL;
   } else {
           goto Err;
   }

   if (REQ_RSYNC == SSHCmd) {
           pw = SYNOChgValForRsync(pw);
   }
   if (!SSHCanLogin(SSHCmd, pw)) {
           goto Err;
   }
   goto Pass;

 Err:
   fprintf(stderr, "Permission denied, please try again.\n");
   exit(1);

 Pass:
   #endif /* MY_ABC_HERE */
...
}

As you can imagine, the IsAllowShell(pw) was the culprit:

static int IsAllowShell(const struct passwd *pw)
{
     struct passwd *pUnPrivilege = NULL;
     char *szUserName = NULL;
     if (!pw || !pw->pw_name) {
             return 0;
     }
     szUserName = pw->pw_name;
     if(!strcmp(szUserName, "root") || !strcmp(szUserName, "admin")){
             return 1;
     }
     if (NULL != (pUnPrivilege = getpwnam(szUserName))){
             if (!strcmp(pUnPrivilege->pw_shell, "/bin/sh") || 
                     !strcmp(pUnPrivilege->pw_shell, "/bin/ash")) {
                     return 1;
             }
     }
     return 0;
}

No wonder why I was experiencing such an odd behavior. Only shells /bin/sh and /bin/ash would be accepted for users different than root or admin. And this regardless of the uid (I had tested also making joeuser uid=0, and it didn't work. Now it's obvious why).

Once identified the cause, the fix was easy: just remove the call to IsAllowShell(). It took me a while to get the right configuration to cross-compile openssh and all its dependencies, but it worked well in the end.

If anyone is interested in doing the same (or trying to cross-compile other kernel modules or binaries for Synology), here's my version of Makefile. It was tested with OpenSSH-5.8p1 source, and works well with models running Marvell Kirkwood mv6281/mv6282 CPU (like DS212+). I used a host running Ubuntu 12.10 x64.

Bottom line: bad practice, terrible code, and a great example of what not to do. I understand sometimes OEMs need to develop special customizations, but they should think twice before digging too deep. Not only this results in unmaintainable code for them, but also creates all sorts of unforeseen issues down the road. Thankfully GPL exist to keep them honest - and open.

Solution 2:

To circumvent the problem, and since I installed bash via ipkg and I can't be sure /opt will always be available (mounted correctly), I simply put the following in my .profile

[ -x /opt/bin/bash ] && exec /opt/bin/bash

while /etc/passwd contains /bin/ash as shell.

Solution 3:

Let's see. It's isolated to a single shell, plus you're looking at sshd debug output, so it's not world writable permission problems with ~joeuser/.ssh. That's the one that gets most people.

Have you tried creating an additional normal user (i.e. not joeuser) to make sure it experiences the same problem? That would isolate it to the user's configuration vs. system-wide configuration.

If it's a system-wide issue, the next thing I'd take a look at are shared configuration files like /etc/profile that get sourced by everyone. There might be a conditional block that isn't firing if the username is root. (not effective userid, since you already tested for that)

Check dmesg for segmentation fault reports if you haven't already, just in case there's something even weirder going on.

Solution 4:

try /etc/ssh/sshd_config
search for AllowUsers

if it's there try adding joeuser there, just username

also it may be blocked in pam... I do not recall which file is it...