Connection closed by UNKNOWN port 65535 when SSH using AD Creds on RHEL Machine

The super-unhelpful ssh error Connection closed by UNKNOWN port 65535 can be reported when your ssh client in a couple of different situations when the remote sshd cannot be reached at all because of something happening "in the middle".

This can be extra-tricky to debug because in some cases the remote sshd has no idea that anyone is ever tried to connect to it.

(Aside 65535 is "special" number to computer folks as it is 2^16 - 1, aka 0xFFF -- the maximum unsigned 16 bit integer (also the max port number))

Case A -- (From @doug 's original question) - In this case the remote sshd got the incoming connection and delegated auth down to Linux libraries for PAM (Pluggable Authentication Modules). PAM hands off to KRB5 or SSS and that fails. So all the poor remote sshd gets is a big NOPE from PAM. ...it never got into it's "normal" protocol parsing and error checking that would let it return a more helpful error message.

(It's possible that old Kerberos config options like gssapiauthentication might behave similarly)

Case B -- In our case we saw this when network firewalls prevented connections from the dev/test machines to staging/production machines. Depending on your network, you might be able get more diagnostics info with tcping $remote_hostname 22, or (less helpful): UDP network tests like ping $remote_hostname, traceroute $remote_hostname, or the IPv6 versions of those commands. Your local network engineers can help confirm & fix.

The giveaway in this case is that ssh -vvv $remote_hostname gets to this point:

debug1: identity file /home/ddickinson/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.6

...pauses for 60s (or whatever timeout), then:

kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

Case C -- Some kinds of failures of the ProxyCommand that your local ssh delegates to can also fail in unhelpful ways. Check for any "proxy*" or "tunnel*" related options in the output of:

ssh -G $remote_hostname