EC2 AMI SSH broken due to home folder permissions

Oy. It's Monday.

I ran these commands so php would have access to some files a cron was writing to the home folder:

chown ec2-user:apache /home/ec2-user
chmod 2775 /home/ec2-user
# the owner of /home/ec2-user used to be root:root.  I intended to change to root:apache, but copy/pasted this line from somewhere and didn't proofread.

Things continued ok until my connections have closed and now I can't SSH into the server anymore. Error is Permission denied (publickey,gssapi-keyex,gssapi-with-mic)

Any ideas to regain access to the server?

This is an Amazon Linux 2 instance on AWS.

Solution 1:

Here's how I solved it (thanks to the System Manager comment by @MisterSmith):

• Visit "AWS System Manager" > Automation

• Search for AWSSupport-TroubleshootSSH and select it.

• Pick the instance (I had to change the "Show managed instances" selection to "Show all instances)

• Change action to "Check All" and Allow Offline to "True" and then Execute

• (A stop/start of the instance was performed as part of the Execution)

Result Output:

----------[Diagnostic Results]----------

module run/openssh               [SUCCESS] All configuration checks passed or all detected problems fixed.
                                 -- FIXED       Permission mode includes write for groups and/or other users: /home/ec2-user
                                 -- FIXED       Missing authorized key directory: /home/ssm-user/.ssh
                                 -- FIXED       Missing authorized key file: /home/ssm-user/.ssh/authorized_keys
                                 -- FIXED       Permission mode includes permissions for groups and/or other users: /etc/ssh/ssh_host_rsa_key
                                 -- FIXED       Permission mode includes permissions for groups and/or other users: /etc/ssh/ssh_host_ecdsa_key
                                 -- FIXED       Permission mode includes permissions for groups and/or other users: /etc/ssh/ssh_host_ed25519_key

I tested SSH access and everything is working as before!

Curiously, the home folder owner was unchanged. Something else in the above output fixed things.

ls -hal of /home/ec2-user in broken state:
drwxrwsr-x 7 ec2-user apache   4.0K May 18 06:59
ls -hal in repaired state:
drwxr-sr-x 7 ec2-user apache   4.0K May 18 06:59