Resume command running in dropped SSH session

Reading this question got me to wondering. Assuming screen is not being used. If an SSH session on a Linux target is dropped, for whatever reason, and you reconnect before the server kills the session because of timeout, is it possible to regain control of the running command such that it will not be aborted because of the broken session?

Solution 1:

Attempting to connect a new terminal's current STD* file descriptors to an old running process is just asking for trouble. Even if you do manage to do that, the terminal's job control won't work as expected. You'll have a mess left behind if you eventually exit the taken-over program, and what happens to the shell that sacrificed its file descriptors to be handed to the newly-backgrounded process. Will ssh stay open when that shell goes away? Probably not. So you'll need to redirect it somewhere else first.

Possible or not, I'd wager that it's more desirable to just let the abandoned process get killed "naturally". If you're doing anything important enough to justify trying to do all the hackery required to resume control and you're on an unstable link, you should probably know that in advance and just use screen (or vnc, or whatever floats your detached-control boat). :)

Solution 2:

I know this is an old question , but I felt it is important to add my findings in case someone else comes across this like I did.

I haven't seen any unusual consequences to doing this yes, but this is what I used and it worked amazingly. Sometimes when we run long processes on our server it will occasionally disconnect the ssh session. The process along with the tty session appears to stay running but we can't reconnect to it. I found the program below to pull the process to the newly connected session.

https://github.com/nelhage/reptyr

Here's more info

https://blog.nelhage.com/2011/02/changing-ctty/

Solution 3:

Generally, the right way to handle this is to prepare for it ahead of time, using GNU screen or bash's nohup or disown mechanisms. If you are using tcsh, the shell will disown background jobs when it exits abnormally.

If you aren't using screen but have managed to keep your process running via one of the disown methods, you might be able to fake reconnecting to the process with gdb (source):

[...] with some dirty hacks, it is not impossible to reopen a process' stdout/stderr/stdin. [...]

And then use gdb for instance to attach to the process, do some call close(0)
call close(1)
call close(2)
call open("/dev/pts/xx", ...)
call dup(0)
call dup(0)
detach

Now, you'd have to tweak this process for your situation. I doubt it would help if you haven't managed to disown the process. If you're using bash, see this post about making bash automatically disown background processes on exit (basically, turn off huponexit with shopt). With a foreground process, you need to have used nohup.

Solution 4:

Probably not. I cannot guarantee that it is impossible but I really doubt it.

One thing is the lack of killing the shell and possible commands running as a consequence of the termination of the ssh connection. This is not so difficult, you should be able to use nohup and similar mechanisms like mentioned in the other question.

But then, assume that you started ssh somehost nuhup vim /some/file and the connection dies. You run ssh somehost to log in again and can see that your vim process is still running. But so, how do you connect to that process again? Interactive forground processes have a controlling tty and the one opened for your vim process when it started would since then have been closed. I am not sure if there is any way of "reopening" it again in your new shell (just as if you have several background jobs running in one shell you cannot foreground any of those in another shell).

Screen have explicitly been written to have this functionality. At startup it forks two processes, a terminal management process and and a client process. The interaction is client <--> terminal manager <--> application, and when you detach or lose connection the client process dies while the terminal manager continues to live. Screen have some specific support to attach to the terminal management process again later on, and I do not think this is possible in the general case.

Solution 5:

retty might be able to help you, but the disclaimers are very real and relevant :)