SSH connection using public/private key authentication fails only for port 22

I have been struggling for hours to make sense of a specific failure from connecting to an SSH server from one recent macOS system.

ssh user@machine -p 22

I have added all keys to ~/.ssh/authorized_keys on the target machine. The keys have been generated with default settings for ssh-keygen.

The server is a freshly provisioned Ubuntu 18.04 instance on Azure, we also tried a different cloud provider.

I can successfully connect with public/private key authentication (PKA) from an Ubuntu machine. I can't connect to port 22 with public/private key authentication (PKA) from the macOS machine in question.

What am I missing?

Symptoms

For connecting from the macOS machine we observe the following symptoms:

Connection to the server works with password authentication
Connection to the server does not work with PKA
When I start a second SSH server using /usr/sbin/sshd -d -p 2222, connection to the server works with PKA (using ssh -p 2222 on the client)
Same for ports 23, 80
Same behavior using
- the built-in SSH client
- a freshly brewed one (version 8.1p1 with brew install openssh)
- cyberduck

Client logs

I checked the logs of ssh -vvv for both ports. The only difference (apart from host key differences) are the following lines which are missing when using port 22 (this makes it very hard to search for the problem):

debug3: put_host_port: [machine]:2222\
...
debug3: put_host_port: [ip-address]:2222\
debug3: put_host_port: [machine]:2222\
...
debug3: receive packet: type 7\
debug1: SSH2_MSG_EXT_INFO received\
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>\

It appears that, when connecting to port 22, the server fails to send (or the client fails to receive) a "packet of type 7" with SSH2_MSG_EXT_INFO and kex_input_ext_info.

Client config

The output of ssh -G (which shows the effective configuration) is identical for port 22 and for other ports.

Server logs

When looking at the output of /usr/sbin/sshd -ddd, the following entry is missing for port 22:

debug3: send packet: type 7 [preauth]

The server doesn't send a "packet type 7"; but perhaps this is just an unrelated symptom?

I also see the following differences:

Port 22:

debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1 [preauth]
debug2: host key algorithms: [email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa [preauth]

Port 80:

debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c [preauth]
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa [preauth]

Extra entries (compared to port 22):

KEX algorithms: ext-c-info
host key algorithms: [email protected],[email protected]

Different order:

ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,

Workaround

For machines that are under my control, I can change the default port of the SSH daemon.

Question

What else can I try to narrow down the problem?

The problem here seems to be that you have "something" interfering with your network traffic that is specifically blocking out your attempt to authenticate via keys on port 22.

This "something" could be so called "endpoint protection" software (software firewall), it could be a network device on your network (i.e. router with deep packet inspection, firewall, etc.) or it could be a network device on the network path or at the server end.

This seems to be the most logical conclusion as this can explain why it doesn't work on port 22, but do work on other ports.

I.e. if you hadn't included the port information, I would have looked at the authorized_keys file on the server. It is possible to specify that a specific key can only be used for authentication when connecting from a specific host - or that it must not be used when connecting from a specific host. If you had set that up to allow the Ubuntu host and not the Mac host - that could explain parts of what you're seeing. However, this method does not include matching again the port number - so it cannot explain that.

This is all assuming that you're running the exact same configuration for port 22 and port 2222 (which you stated you are), and that you're running standard OpenSSH software on the server. If you're not, that software might include configuration to discriminate based on ports - but I don't think it is likely.