Persistent reverse SSH tunnels

We have a number of servers (currently in the 10s, soon the low 100s) deployed to varying institutions with varying network layouts. To make sure that we can access the servers at will we have them setting up reverse SSH tunnels back to a server we have in "the cloud" solely for this purpose. To keep the connections open persistently the guys who set it up used autossh and have supervisor monitoring it. The problem we're experiencing is that periodically (and somewhat frequently) the connections will just close mid-use... as in, you can be actively typing and BAM Connection to localhost closed.

If anyone could help me sort out what I'd need to change to get the connections to be more stable, I'd greatly appreciate it. If you need to see any other config files, just ask.

The main supervisor config file looks like:

[unix_http_server]
file=/var/run//supervisor.sock
chmod=0700

[supervisord]
logfile=/var/log/supervisor/supervisord.log
pidfile=/var/run/supervisord.pid
childlogdir=/var/log/supervisor

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run//supervisor.sock

[include]
files = /etc/supervisor/conf.d/*.conf

The supervisor config file specifically for the autossh is:

[program:autossh]
command=autossh -M29001 -f -N -R 10033:localhost:22 [email protected]
user=root
autostart=true
autorestart=true
stopsignal=KILL

Solution 1:

You'll get a much more reliable setup by doing it properly with an VPN server on your central jumpbox server and having the customer machines VPN into that server and get an IP.

Solution 2:

I've had IDLE SSH tunnels time out regularly. To solve that, I pass -o "ServerAliveInterval=10" to ssh (which should also work with autossh). However, since you report that this can occur to an active SSH session, this may not actually fix your problem.