SSH multi-hop connections with netcat mode proxy

Since OpenSSH 5.4 there is a new feature called natcat mode, which allows you to bind STDIN and STDOUT of local SSH client to a TCP port accessible through the remote SSH server. This mode is enabled by simply calling ssh -W [HOST]:[PORT]

Theoretically this should be ideal for use in the ProxyCommand setting in per-host SSH configurations, which was previously often used with the nc (netcat) command. ProxyCommand allows you to configure a machine as proxy between you local machine and the target SSH server, for example if the target SSH server is hidden behind a firewall.

The problem now is, that instead of working, it throws a cryptic error message in my face:

Bad packet length 1397966893.
Disconnecting: Packet corrupt

Here is an excerpt from my ~/.ssh/config:

Host *
  Protocol 2
  ControlMaster auto
  ControlPath ~/.ssh/cm_socket/%r@%h:%p
  ControlPersist 4h

Host proxy-host proxy-host.my-domain.tld
  HostName proxy-host.my-domain.tld
  ForwardAgent yes

Host target-server target-server.my-domain.tld
  HostName target-server.my-domain.tld
  ProxyCommand ssh -W %h:%p proxy-host
  ForwardAgent yes

As you can see here, I'm using the ControlMaster feature so I don't have to open more than one SSH connection per-host.

The client machine I tested this with is an Ubuntu 11.10 (x86_64) and both proxy-host and target-server are Debian Wheezy Beta 3 (x86_64) machines.

The error happens when I call ssh target-server. When I call it with the -vvv flag, here is what I get additionally:

OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011
debug1: Reading configuration data /home/aef/.ssh/config
debug1: Applying options for *
debug1: Applying options for target-server.my-domain.tld
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Applying options for target-server.my-domain.tld
debug1: auto-mux: Trying existing master
debug1: Control socket "/home/aef/.ssh/cm_socket/[email protected]:22" does not exist
debug2: ssh_connect: needpriv 0
debug1: Executing proxy command: exec ssh -W 192.0.2.195:22 gateway-host.my-domain.tld
debug1: identity file /home/aef/.ssh/id_rsa type -1
debug1: identity file /home/aef/.ssh/id_rsa-cert type -1
debug1: identity file /home/aef/.ssh/id_dsa type -1
debug1: identity file /home/aef/.ssh/id_dsa-cert type -1
debug1: identity file /home/aef/.ssh/id_ecdsa type -1
debug1: identity file /home/aef/.ssh/id_ecdsa-cert type -1
debug1: permanently_drop_suid: 1000
Host key fingerprint is 1a:2b:3c:4d:5e:6f:7a:8b:9c:ad:be:cf:de:ed:fe:ef
+--[ECDSA  521]---+
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
|                 |
+-----------------+

debug1: Remote protocol version 2.0, remote software version OpenSSH_6.0p1 Debian-3
debug1: match: OpenSSH_6.0p1 Debian-3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-7ubuntu1
debug2: fd 5 setting O_NONBLOCK
debug2: fd 4 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "192.0.2.195" from file "/home/aef/.ssh/known_hosts"
debug3: load_hostkeys: loaded 0 keys
debug3: load_hostkeys: loading entries for host "192.0.2.195" from file "/etc/ssh/ssh_known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /etc/ssh/ssh_known_hosts:49
debug3: load_hostkeys: found key type RSA in file /etc/ssh/ssh_known_hosts:50
debug3: load_hostkeys: loaded 2 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
Bad packet length 1397966893.
Disconnecting: Packet corrupt

Update: Now with -vvv instead of only -v output.

Solution 1:

I finally found out what this is all about. It seems to be a bug in OpenSSH when ControlMaster is on for both the proxy-host and the target-server. There are however at least these two work-arounds:

• Make sure there is already a running connection to the proxy-host before trying to connect to the target-server. This will make the error vanish and everything works as expected. You can do that by manually connecting to proxy-host.

• Disable ControlMaster for the ProxyCommand like ProxyCommand ssh -o "ControlMaster no" -W %h:%p proxy-host. This will also get rid of the problem, but it will create a new connection to proxy-host for every connection using that exact same ProxyCommand.

Solution 2:

The real problem here is the ControlPersist since this option appeared in OpenSSH 5.6.

You should consider upgrading your openssh server to >=5.6 or simply remove the directive from your client config file.

Regards