Did Mojave break SSH login?

I hope someone can help me troubleshoot this ssh login issue. In my home LAN, I develop websites and have set up my router to forward external SSH calls to my main laptop from this external port.

So if I'm logged into a customer's computer I would ssh, or scp

$ ssh -p 7219 sam@mycomcastip

which worked well until a few weeks ago when I upgraded it to Mojave. Now the connection times out. Here's my steps to isolate the issue:

$ ssh -p 7220 sam@mycomcastip

works because my router port forwards it as port 22 to an older machine running High Sierra. Maybe my router is goofed? I thought so, so I installed OpenWRT on it (which is very good, so far) and I cannot ssh into the laptop from outside as I could before.

Perhaps Comcast is blocking port 7219? It if was, it would be blocking port 7220 and it's not blocking it. Changing the port forward to another port fails with the same timeout.

Perhaps it's my SSH daemon that's goofed? Changing the port forward to route to the laptop's port 80 (which has Apache running), instead of 22 yields no difference in the result.

I can confirm that inside my LAN I can SSH into this laptop with no issues. That assures me that the laptop's ssh daemon is running. I have no trouble ssh out from this machine.

I can confirm the router is not doing the blocking because I can run tcpdump on the router looking for this port and it receives it and passes it on.

I've run tcpdump on this laptop and examining the PCAP in Wireshark I can see the packet coming in on port 22 and subsequent attempts by the remote operator retrying the connection, but I just haven't found the right tutorial on analyzing a packet for this issue to help me see what might be the cause.

41 3.349934 remoteIP 192.168.1.105 TCP 74 45166 → 22 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1415111467 TSecr=0 WS=512

Perhaps someone can shed light on what this means?

I attempted to install Mojave on another machine to see if it is a Mojave issue but every other machine is too old to run it so I'm unable to isolate that it's the hardware or software on this machine. Speaking of which, this behavior occurs when I'm routing to the laptop through the wireless and Ethernet/Thunderbolt of the laptop which would suggest that if it's hardware, it's not the networking ports causing the issue.

Since the port forwarding rules are the same except for the source and destination ports, I'm leaning on thinking something changed in Mojave as that's the only change I've recall that I made to the laptop when this capability broke.

Looking online, I found the closest which suggests that Apple has closed off certain folders to remote users. Except as this poster determines, SSH gets around this block so that may not be it.

https://www.sentinelone.com/blog/mojaves-security-hardening-user-protections-bypassed/

I discovered a clone of this machine on High Sierra before I upgraded it to Mojave. Running off that clone, the expected behavior occurs. Running tcpdump on this laptop I can see the TCP passed through the router to this laptop from the remote server and I can see my laptop is ACK in return, which confirms what I saw when I SSH in from remote. That seems to eliminate that it's my firewall on the router that is blocking any of these odd ports to this laptop. Which points more likely that Mojave is blocking port 22 from outside its subnet.

I can show screen cap that shows this Mojave firewall is turned off if important.

That's why I ask here, instead of another networking forum as it seems to be only the Mojave OSX machine that is rejecting port 22. What other troubleshooting steps should I try?

Solution 1:

There's a key statement in your question that jumps out:

I can confirm that inside my LAN I can SSH into this laptop with no issues. That assures me that the laptop's ssh daemon is running. I have no trouble ssh out from this machine.

This confirms that it can't be a Mojave issue. Mojave is accepting SSH connections. However, that doesn't rule out other services running on the Mojave machine.

Your testing with tcpdump and setting up a different machine with port forwarding rules confirms that it's not the router.

Your comment re: running in Safe Mode led you to the VPN client software which seemed to be the culprit.

VPN software (per it's configuration) can redefine the gateway for traffic. Based on what you've described, it sounds like traffic bound for port 22 was properly being forwarded, but the VPN software was responding via the VPN gateway, not back to the router, thus the time out.

Solution 2:

1. SSH key bit length

Is your SSH key bit length > 2048? You can use this command to confirm.

$ ssh-keygen -lf ~/.ssh/id_rsa.pub
4096 SHA256:0f7e9153ec1edf81c224fec24c76d3ab1be7010e [email protected] (RSA)

If it's less macOS will refuse to allow it.

2. Cipher Suite Support

Also you should investigate from the client where you're running SSH which cipher suites are being presented to your laptop's SSH server. You can do this using ssh -vvvv .... to see what ciphers are available on your client like so:

$ ssh -Q cipher
3des-cbc
aes128-cbc
aes192-cbc
aes256-cbc
[email protected]
aes128-ctr
aes192-ctr
aes256-ctr
[email protected]
[email protected]
[email protected]

You'll also want to investigate several other components of the cipher suite.

Ciphers: ssh -Q cipher
MACs: ssh -Q mac
KexAlgorithms: ssh -Q kex
PubkeyAcceptedKeyTypes: $ ssh -Q key

The man page's details on -Q as well:

 -Q query_option
         Queries ssh for the algorithms supported for the specified version 2.  The available features
         are: cipher (supported symmetric ciphers), cipher-auth (supported symmetric ciphers that sup-
         port authenticated encryption), mac (supported message integrity codes), kex (key exchange
         algorithms), key (key types), key-cert (certificate key types), key-plain (non-certificate
         key types), and protocol-version (supported SSH protocol versions).

3. IPQoS

There have been reports from newer versions of OpenSSH pertaining to connectivity issues. I've found these as examples:

SSH fails on macOS Mojave 10.14.1
OpenSSH 7.8 causes VMware Workstation and Player networking SSH bug on Linux VMs #287
SSH connections dropped due to QoS flags

The workaround requires adding the following to your ~/.ssh/config:

$ cat ~/.ssh/config
...
...
Host *
  IPQoS throughput

I've seen variations of this so you may need to try low instead of throughput. You can consults the man ssh_config for more details, here's the excerpt for that option:

 IPQoS   Specifies the IPv4 type-of-service or DSCP class for connections.  Accepted values are af11,
         af12, af13, af21, af22, af23, af31, af32, af33, af41, af42, af43, cs0, cs1, cs2, cs3, cs4,
         cs5, cs6, cs7, ef, lowdelay, throughput, reliability, a numeric value, or none to use the
         operating system default.  This option may take one or two arguments, separated by white-
         space.  If one argument is specified, it is used as the packet class unconditionally.  If two
         values are specified, the first is automatically selected for interactive sessions and the
         second for non-interactive sessions.  The default is af21 (Low-Latency Data) for interactive
         sessions and cs1 (Lower Effort) for non-interactive sessions.

You can of course pass it via CLI like so as well:

$ ssh -o IPQoS=throughput [email protected]

If find you're not having any luck with the above you may want to try this form instead:

Host *
     IPQoS lowdelay throughput

Several threads mention this form working as well.

References

ssh man page