"Cannot set terminal process group" during su to another user as login shell

Solution 1:

• su - username is interpreted by your su to mean "run username's shell as an interactive login shell"
• su username - is interpreted by your su to mean "run the following non-interactive command (-) as username"
• the latter only worked at all because:
  • your su passes trailing arguments to sh for parsing
  • sh takes - to mean "run as a login shell (read /etc/profile, ...)"

But what you're really interested in is: why non-interactive? Sharing the controlling terminal between the privileged parent and the unprivileged child leaves you vulnerable to "TTY pushback privilege escalation", aka the TIOCSTI bug, so unless you really need it su detaches from it. When you used the su username - form, su inferred that you didn't need a controlling terminal.

Only processes with a controlling terminal can have session leaders which manipulate process groups (do job control); the trace you gave is bash detecting that it can't be a session leader.

You mention:

Where it gets stranger is that both forms work fine on Ubuntu and CentOS 6, however on vanilla Debian, only the first form works without error.

Ignoring variants like sux and sudo, there are at least three^[1] versions of su on Linux: coreutils, util-linux and shadow-utils from which Debian's comes. The latter's manpage points out:

This version of su has many compilation options, only some of which may be in use at any particular site.

and Debian's comes with the flag old_debian_behavior; other versions may have similar compile-time/runtime options. Another reason for variability might be that there was some debate^[2] as to whether su should ever be used to drop privilege this way and whether the TIOCSTI bug is therefore a bug at all (Redhat originally closed it "WONTFIX").

[1]: Edit: add SimplePAMApps and hardened-shadow to that.

[2]: Solar Designer has some (old) opinions there which I think are worth a read.

Solution 2:

I would check the ownership and permissions on /dev/pts* or for a new configuration for udev related to /dev/pts devices, that was not replaced during the upgrade process.

You can also try to find out what syscal is generating the error by running as root:

strace -f su - username 2>stderr.log