Setting up sftp on Amazon Linux 2 with ssh keys, user segregation (sftp vs ssh), different ports, and user directory constraints

TDLR: I have a Catch 22 where, depending on permissions on the user's home directory, I can get the SSH authentication to work, or the user directory constraints, but not both.

BTW, I really want to roll my own SFTP server. Please don't recommend I try AWS Transfer service or something alternative. Thanks.

Here is relevant (changed from default) content in /etc/ssh/sshd_config:

Subsystem sftp internal-sftp
Port 22
Port 2299
Match Group sftpusers LocalPort 2299
  ChrootDirectory /sftp-data/%u
  ForceCommand internal-sftp
Match Group sftpusers LocalPort 22
  DenyGroups sftpusers
Match LocalPort 2299  Group *,!sftpusers
  DenyUsers *

I want port 22 functioning as ssh typically does, but only for non-sftp users. For sftp users, in group "sftpusers", I want port 2299 to function as sftp only, and not ssh. For non-sftpusers, I want access to port 2299 denied.

Ok, so, I created a user "user1" with home directory /sftp-data/user1 and shell /sbin/nologin. I created /sftp-data/user1/.ssh/authorized_keys and populated that with the public ssh key. /sftp-data is owned by root with 700 permissions. /sftp-data/user1/.ssh and below are owned by user1, and /sftp-data/user1/.ssh/authorized_keys has permission 600. The ownership/permissions of /sftp-data/user1 are under question here. More below.

I created the user group sftpusers and added user1 to that group. The built-in ec2-user you get with AWS is not a member of that group, however. Testing with ec2-user worked great: access via ssh, port 22 worked as it always does, but no access to port 2299.

So, testing with user1 is where it got interesting. User1 has no access to port 22 - that's great! With user1 owning /sftp-data/user1, the ssh public key authentication succeeds on port 2299, but the user is immediately logged out, with this message saved in /var/log/secure:

Sep  2 19:21:38 ip-192-168-0-25 sshd[10369]: Accepted publickey for user1 from <ip-address redacted> port 61110 ssh2: ECDSA SHA256:<sha redacted>
Sep  2 19:13:23 ip-192-168-0-25 sshd[9803]: pam_unix(sshd:session): session opened for user user1 by (uid=0)
Sep  2 19:13:23 ip-192-168-0-25 sshd[9803]: fatal: bad ownership or modes for chroot directory "/sftp-data/user1" [postauth]
Sep  2 19:13:23 ip-192-168-0-25 sshd[9803]: pam_unix(sshd:session): session closed for user user1

Sure, that makes sense. Chroot requires /sftp-data/user1 to be owned by root, permissions 700. So, make that so, and now the sftp (ssh key) authentication fails.

Sep  2 19:41:00 ip-192-168-0-25 sshd[11693]: error: AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys user1 SHA256:<sha redacted> failed, status 22

BTW, eic_run_authorized_keys is a wrapper AWS puts around standard ssh authentication to enable AWS Instance Connect.

For extra credit... if the above problem is not challenging enough, can you come up with a scheme where I can give particular sftp users access to particular project directories, and only those, without creating a group for every project? Link from each user's home directory to a project directory would be awesome.

Additional info requested by @anx:

# getent passwd user1
user1:x:1001:1001::/sftp-data/user1:/sbin/nologin
# namei -l /sftp-data/user1/.ssh/authorized_keys
f: /sftp-data/user1/.ssh/authorized_keys
dr-xr-xr-x root  root      /
drwxr-xr-x root  root      sftp-data
drwx------ root  root      user1
drwx------ user1 sftpusers .ssh
-rw------- user1 sftpusers authorized_keys

I turned on DEBUG logging for sshd. Using the ChrootDirectory directive, with /sftp-data/user1 owned by root, and SSH authentication failing, I see this in /var/log/secure:

debug1: Could not open authorized keys '/sftp-data/user1/.ssh/authorized_keys': Permission denied

ps clearly shows me that root is running the sshd process.

Your ChrootDirectory is /sftp-data/user1, which must be owned by root. And it is:

# namei -l /sftp-data/user1/.ssh/authorized_keys
f: /sftp-data/user1/.ssh/authorized_keys
dr-xr-xr-x root  root      /
drwxr-xr-x root  root      sftp-data
drwx------ root  root      user1
drwx------ user1 sftpusers .ssh
-rw------- user1 sftpusers authorized_keys

However, at this point the sshd has already changed the user to user1 and dropped privileges, so user1 cannot descend into lower level directories. To do that the directory needs search permission (a+x).

chmod a+x /sftp-data/user1

Now user1 can descend to subdirectories and the .ssh directory should be readable.

I think this is primarily a folder permissions issue. I think that /sftp-data/user1/.ssh and the directories below it must be owned by user1, but it sounds like they are owned by root. Try the following as root:

Verify the public key in /sftp-data/user1/.ssh/authorized_keys is accurate. After that, I think these are the permissions you may need to change:

chmod 700 /sftp-data/user1/.ssh
chmod 600 /sftp-data/user1/.ssh/authorized_keys
chown root:sftpusers /sftp-data
chown -R user1:user1 /sftp-data/user1/.ssh

Restart SSH and I think you'll be good to go.