Remote code execution after tmux reboot

I've read about tmux-resurrect, but it's still not clear to me if I can do the following:

1. start tmux session
2. ssh to remote server
3. launch long-running code on remote host
4. kill tmux server (e.g. local machine reboot)
5. restore & reattach tmux session (e.g. using tmux-resurrect)
6. continue monitoring the code I launched on the remote server

Do I need to use nohup in step 3? or can tmux-resurrect (or some other tool) take care of it?

Solution 1:

No local-side tool can do this, because of what the server "sees":

• if you disconnect SSH gracefully, the server will notice this immediately;
• if you just disappear (e.g. by a hard reset), the connection will stay open on the server side until some kind of timeout (compare ClientAlive* options, TCP keepalive concept – see this answer for some details).

Even if you intended to use tmux-resurrect or another tool on the local side, the server doesn't know, doesn't care. It will terminate your long-running code unless you used nohup or better…

If you can, use tmux (or screen) on the server:

1. Don't enter tmux session on the client.
2. ssh to the server.
3. Start tmux session on the server.
4. Launch long-running code there.
5. Disconnect anyhow: gracefully or not, with or without detaching from the remote tmux.
6. ssh again.
7. Your tmux session is still there (unless something bad happened to the server in the meantime); reattach with tmux a.

I use tmux this way on daily basis and I think this is the right way. On my laptop tmux sessions last for few hours at most, I shut it down every night; but I have access to a couple of servers where my tmux sessions run for months. When I'm writing this, on one of them there's watch df -h process running literally for a week in a tmux session started like two months ago.