SSH: Troubleshooting "Remote port forwarding failed for listen port" errors

Question: Why does ssh -N -R 2222:localhost:22 <bluehost_user>@<bluehost_ip> result in a "Remote port forwarding failed for listen port" error? The objective is to establish a reverse tunnel with port forwarding in order to consistently ssh into a host behind a NAT router that has a dynamic private IP. See image for details.

Already Tried:

1. Researched existing literature on Google, Stackoverflow, etc. There are topics concerning this error message, however the resolutions given resolve root causes different than that of this particular instance because those resolutions do not resolve the error in this case.
2. I've performed several diagnostics to validate the required ports are open. Some of those results are shown in the image below.

Reverse SSH Tunnel

Update

I was trying the following command for Step 2: reduser@redhost:~ ssh greenuser@greenhost -p 2222

It should be: reduser@redhost:~ ssh greenuser@bluehost -p 2222 You want to use the greenuser credentials on the bluehost IP because the host you are loging into when you use port 2222 is really the greenhost.

Solution 1:

Why does ssh -N -R 2222:localhost:22 <bluehost_user>@<bluehost_ip> result in a "Remote port forwarding failed for listen port" error?

I get this exact warning when I attempt to use a port that is already taken on the remote side.

The output of netstat from bluehost indicates that something is already listening on port 2222 there. It doesn't show what it is though.

Solutions:

1. Change 2222 in your ssh invocation to some other port which is not in use on bluehost. Just make it greater than 1023 because regular users can't bind to well-known ports; otherwise you will get the same warning regardless of whether the port is in use or not.
2. Or identify the listening process (on bluehost) with sudo lsof -i TCP:2222; terminate or reconfigure it to make the port 2222 available.

Edit:

In your case this part of man ssh seems important:

-R [bind_address:]port:host:hostport
-R [bind_address:]port:local_socket
-R remote_socket:host:hostport
-R remote_socket:local_socket

[…] By default, TCP listening sockets on the server will be bound to the loopback interface only. This may be overridden by specifying a bind_address. An empty bind_address, or the address ‘*’, indicates that the remote socket should listen on all interfaces. Specifying a remote bind_address will only succeed if the server's GatewayPorts option is enabled (see sshd_config(5)).

It means you should have GatewayPorts yes in the sshd_config on bluehost. Read man 5 sshd_config to learn more. Don't forget to reload the service afterwards.

Solution 2:

Make sure there is no hanging connection on port 2222 at bluehost. Test at bluehost lsof -t -i:2222 whether any process id is using port 2222. Additionally, kill this process (for example with kill $(lsof -t -i:2222)).

This resolved the issue for me. Hopefully this information is useful for someone else. :)