Is it possible to SSH access a server with a misconfigured subnet?

We have a server where one of our engineers misconfigured the subnet and now we are locked out of this server and the only access that I know of would work is a serial console from IDC (it means asking an IDC engineer to help us with this).

What has been misconfigured:

address 192.168.1.9 # Original address there was
netmask 255.255.255.254 # Misconfigured, originally should've been .240

Out of curiosity - is there a way to avoid calling IDC and somehow connect to this host over SSH (then we can fix the configuration)?

Solution 1:

You need to be able to log in on another host on the same network segment. Some of the ways to get access to the misconfigured host requires root on the intermediate host, but there also is one easy way to get access without needing root on the intermediate host.

The easy way to access the host using IPv6

ssh -o ProxyCommand='ssh -W [fe80::42:ff:fe:42%%eth0]:%p user@intermediate-host' root@target-server

The following example values in above command need to be substituted with correct values for your use case: fe80::42:ff:fe:42, eth0, user, intermediate-host, and target-server.

Detailed explanation of how it works

ProxyCommand is an ssh feature to use when you cannot open a TCP connection directly to the target host. The argument to ProxyCommand is a command whose stdin/stdout to use instead of a TCP connection.

-W is used to open a single port forwarding and connect it to stdin/stdout. This fits nicely together with ProxyCommand.

fe80::42:ff:fe:42%%eth0 is the link-local address of the target host. Notice that due to ProxyCommand using % as escape character, the typed ssh command must use %% in that location. You can find all link-local addresses on the segment by running ssh user@intermediate-host ping6 -nc2 ff02::1%eth0.

Using IPv6 link-local addresses for this purpose is usually the easiest way because it is enabled by default on all modern systems, and link-local addresses keep working even if both IPv4 and IPv6 stacks are severely misconfigured.

Falling back to IPv4

If IPv6 is completely disabled on the misconfigured host (absolutely not recommended), then you may have to resort to using IPv4. Since IPv4 doesn't have link-local addresses the way IPv6 does then accessing the misconfigured host using IPv4 gets more complicated and need root access on the intermediate host.

If the misconfigured host was still able to use its default gateway, you would be able to access it from outside. Possibly the misconfigured netmask also broke the default gateway due to the stack refusing to use a gateway outside of the prefix covered by the netmask. If this is indeed the case, the misconfigured host will only be able to communicate with 192.168.1.8 because that's the only other IP address in the subnet currently accessible to this misconfigured host.

If you have a login on 192.168.1.8, you might just be able to ssh from there to 192.168.1.9. If 192.168.1.8 is currently unassigned you can temporarily assign it to any host on the segment on which you have root access.

Solution 2:

kasperd posted a great answer detailed enough for me to learn how I can recover from situation in the question. This answer is the exact step-by-step of how I did it.

1. SSH to server on same physical network
2. Using arp -a or ip neighbor list as root find the MAC address of misconfigured server.
3. Using MAC to link-local converter find link-local for misconfigured server
4. Now can SSH to server as any user via ssh user@link-local%dev where:
   • user - username for which we are allowed to SSH
   • link-local - self-assigned IPv6 address recovered in step 3
   • dev is physical interface this server is reachable from (e.g. eth0)