How to detach ssh session without killing a running process?

I started a test filesystem resize on a server through ssh, but totally forgot to do that on a screen session. Now it is running, and my laptop has very slow internet. Is there any way that I can keep it running if I loose the ssh connection?

It also outputs progress, but I don't have to care about getting that.

OS: RHEL7.2 equivalent (SL7).

It's probably better to link this as the solution: https://stackoverflow.com/questions/625409/how-do-i-put-an-already-running-process-under-nohup

don't know how to link that instead of the suggested one though, so feel free to add that and mark as duplicate

Solution 1:

Use reptyr

This is exactly the case man 1 reptyr explicitly mentions:

reptyr is a utility for taking an existing running program and attaching it to a new terminal. Started a long-running process over ssh, but have to leave and don't want to interrupt it? Just start a screen, use reptyr to grab it, and then kill the ssh session and head on home.

(The manual mentions screen, you can use tmux instead, whichever you prefer).

Don't miss this note:

reptyr depends on the ptrace(2) system call to attach to the remote program. On Ubuntu Maverick and higher, this ability is disabled by default for security reasons. You can enable it temporarily by doing

echo 0 > /proc/sys/kernel/yama/ptrace_scope

as root, or permanently by editing the file /etc/sysctl.d/10-ptrace.conf, which also contains more information about this setting.

If the file doesn't exist but the /etc/sysctl.d/ directory does, then it's probably enough to create it with the following content:

kernel.yama.ptrace_scope = 0

The setting will be applied at the next reboot. Please see security considerations below.

Basic usage

Basic usage is simple:

reptyr PID

where PID is the PID of the process you want to attach to a new terminal. Note reptyr only attaches a process to another terminal. This does not mean the process becomes a child of the new shell.

Security considerations

Setting ptrace_scope as 0 is not recommended.

As Linux grows in popularity, it will become a larger target for malware. One particularly troubling weakness of the Linux process interfaces is that a single user is able to examine the memory and running state of any of their processes. For example, if one application (e.g. Pidgin) was compromised, it would be possible for an attacker to attach to other running processes (e.g. Firefox, SSH sessions, GPG agent, etc) to extract additional credentials and continue to expand the scope of their attack without resorting to user-assisted phishing.

This is not a theoretical problem. SSH session hijacking (http://www.storm.net.nz/projects/7) and arbitrary code injection (http://c-skills.blogspot.com/2007/05/injectso.html) attacks already exist and remain possible if ptrace is allowed to operate as before. Since ptrace is not commonly used by non-developers and non-admins, system builders should be allowed the option to disable this debugging system.

[…]

The sysctl settings (writable only with CAP_SYS_PTRACE) are:

0 - classic ptrace permissions: a process can PTRACE_ATTACH to any other process running under the same uid, as long as it is dumpable […]

1 - restricted ptrace: a process must have a predefined relationship with the inferior it wants to call PTRACE_ATTACH on. By default, this relationship is that of only its descendants when the above classic criteria is also met. […]

2 - admin-only attach: only processes with CAP_SYS_PTRACE may use ptrace with PTRACE_ATTACH, or through children calling PTRACE_TRACEME.

3 - no attach: no processes may use ptrace with PTRACE_ATTACH nor via PTRACE_TRACEME. Once set, this sysctl value cannot be changed.

^(source)

A reasonable approach is to set ptrace_scope to 2 and then allow reptyr to use ptrace:

echo 2 | sudo tee /proc/sys/kernel/yama/ptrace_scope
sudo setcap CAP_SYS_PTRACE+pe /usr/bin/reptyr

Don't forget about the file that holds the permanent setting.

Capabilities are stored in the file's inode. Therefore I won't be surprised if reptyr loses the capability when it gets updated (atomically replaced by a new executable).

Solution 2:

Use

Ctrl-Z

to pause the application and return to the command line. Then use

bg

to allow the process to continue in the background. Finally use

disown

So that the process won't be closed when you disconnect your session and close your terminal.

The process will continue to run but there's no way to 'reattach' to the terminal to view the output that I'm aware of if you reconnect to a new session.

Solution 3:

You can totally disassociate a command (in bash or zsh, possibly other shells) with the disown command. The command however may not be happy about this if it requires a tty, in which case see the reptyr answer.

$ ssh somecentos7system
-bash-4.2$ sleep 252727
^Z
[1]+  Stopped                 sleep 252727
-bash-4.2$ bg
[1]+ sleep 252727 &
-bash-4.2$ disown
-bash-4.2$ logout
Connection to somecentos7system closed.
$ ssh somecentos7system
-bash-4.2$ pgrep -lf 252727
20089 sleep
-bash-4.2$ 

Another option is to have a screen (or tmux) session automatically started so you can't forget to start one because one was already started for you because you set things up that way. There's other posts elsewhere on how to do that.