On AWS do I have to open ports in an EC2 instance's firewall as well as security group?

If I change my SSH port from 22 to 23453, I can no longer ssh in.

In more detail, I'm using a Red Hat EC2 instance on Amazon Web Services. This is the second change I've on a fresh install (first change was to add a non-root user).

I can ssh in fine using Git Bash and a local .ssh/config file, I edit the line in /etc/ssh/sshd_config that currently says

#Port 23453

to say

Port 23453

then restart sshd with

sudo service sshd restart

I then add a line "Port 23453" my .ssh/config file

Host foo 
Hostname my-ec2-public-DNS
Port 23453
IdentityFile my ssl key

If I open another Git Bash shell (without closing my existing connection) and attempt to ssh into my instance (with ssh foo) I see the following error:

ssh: connect to host my-ec2-public-DNS port 23453: Bad file number

The security group attached to this instance has two entries, both TCP

22 (SSH) 0.0.0.0/0

23453 0.0.0.0/0

My best guess is that the port is still blocked by my firewall.

The output of sudo iptables -L is as follows

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Which looks pretty open to me.

UPDATE

After adding an iptables rule

iptables -A INPUT -p tcp --dport 23453 -j ACCEPT

and trying again, still no luck.

Output of iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:23453

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Which looks sufficiently open. I'm not entirely sure how to look for packets coming in or activity on the port. But the output of netstat -ntlp (as root)

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State PID/Program name
tcp        0      0 0.0.0.0:56137               0.0.0.0:*                   LISTEN      948/rpc.statd
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      930/rpcbind
tcp        0      0 127.0.0.1:631               0.0.0.0:*                   LISTEN      1012/cupsd
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      1224/master
tcp        0      0 0.0.0.0:23453               0.0.0.0:*                   LISTEN      32638/sshd
tcp        0      0 :::36139                    :::*                        LISTEN      948/rpc.statd
tcp        0      0 :::111                      :::*                        LISTEN      930/rpcbind
tcp        0      0 ::1:631                     :::*                        LISTEN      1012/cupsd
tcp        0      0 :::23453                    :::*                        LISTEN      32638/sshd

Which seems to me to show sshd on 23453.

I've checked again that the instance has the port open in the security group (Port: 23453, Protocol: tcp, Source: 0.0.0.0/0)

What else can be causing the failure to connect via SSH?

Cheers

POSTMORTEM

I can now connect. It was a missing rule in iptables. The output of iptables -L now looks like this:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:23453 state NEW
ACCEPT     all  --  anywhere             anywhere
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:ssh
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Solution 1:

Your instance firewall doesn't have this port open. Try the following command:

iptables -I INPUT 3 -s 0.0.0.0/0 -d 0.0.0.0/0 -p tcp --dport 23453 -m state --state New -j ACCEPT

Note that iptables rules need to be saved to persist after a reboot. On RHEL that's:

/sbin/service iptables save

Solution 2:

Add an iptables rule

iptables -I INPUT 1 -p tcp --dport 23435 -j ACCEPT

which accepts traffic from any host, by port 23435, and try to ssh, if you see any packets or activity, then it means that the packets are reaching your server.

If you see no packets, that means that the AWS Security group does not have rule to allow your port.

but if you see traffic on this rule (by iptables -nvL), then you have to run "netstat -ntlp" and verify if SSH daemon is running on port 2435. and on 0.0.0.0/0.

hopefully these steps would resolve the issue. if still no, then tell me.