How to launch a long-running Python code in a Docker container running on a remote server [duplicate]

Let's say I launch a bunch of processes from a ssh session. Is it possible to terminate the ssh session while keeping those processes running on the remote machine?

You should look for modern alternatives like tmux.

tmux is superior to screen for many reasons, here are just some examples:

• Windows can be moved between session and even linked to multiple sessions
• Windows can be split horizontally and vertically into panes
• Support for UTF-8 and 256 colour terminals
• Sessions can be controlled from the shell without the need to enter a session

Basic Functionality

To get the same functionality as explained in the answer recommending screen, you would need to do the following:

• ssh into the remote machine
• start tmux by typing tmux into the shell
• start the process you want inside the started tmux session
• leave/detach the tmux session by typing Ctrl+b and then d

You can now safely log off from the remote machine, your process will keep running inside tmux. When you come back again and want to check the status of your process you can use tmux attach to attach to your tmux session.

If you want to have multiple sessions running side-by-side, you should name each session using Ctrl+b and $. You can get a list of the currently running sessions using tmux list-sessions or simply tmux ls, now attach to a running session with command tmux attach-session -t <session-name>.

tmux can do much more advanced things than handle a single window in a single session. For more information have a look in man tmux or the tmux GitHub page. In particular, here's an FAQ about the main differences between screen and tmux.

Option 1: nohup

The best way is often the simplest.

nohup long-running-command &

It was made specifically for this, it even logs stdout to nohup.log.

man nohup

Option 2: bg + disown

ctrl+z
bg
disown -h

If you want to "background" already running tasks, then Ctrl+Z then run bg to put your most recent suspended task to background, allowing it to continue running. disown will keep the process running after you log out. The -h flag prevents hangup.

screen and others can do it, but that's not what they're for. I recommend nohup for tasks you know you are going to leave behind and bg for tasks you're already running and don't want to re-start.

Keep in mind, both are bash specific. If you're not using bash, then the commands could be different.

You could do that by using screen.

Type man screen to find out more or read this screen man page.

Simple scenario:

• ssh into your remote box. Type screen Then start the process you want.

• Press Ctrl-A then Ctrl-D. This will "detach" your screen session but leave your processes running. You can now log out of the remote box.

• If you want to come back later, log on again and type screen -r This will "resume" your screen session, and you can see the output of your process.

Screen and nohup is the better way, but if you have to detach a process already running without screen or nohup you can run disown command.

disown [-ar] [-h] [jobspec… |pid… ]

Without options, remove each jobspec from the table of active jobs. If the -h option is given, the job is not removed from the table, but is marked so that SIGHUP is not sent to the job if the shell receives a SIGHUP. If jobspec is not present, and neither the -a nor the -r option is supplied, the current job is used. If no jobspec is supplied, the -a option means to remove or mark all jobs; the -r option without a jobspec argument restricts operation to running jobs.

With disown you can close the terminal and get the process running on the machine.

I was stuck in a large mv so I wasn't in a position to stop the process, setup screen and then start it again. I managed to exit the SSH session with the process running by essentially doing the following steps:

1. Establish SSH connection: ssh user@host
2. Run the desired command to start the process
3. Press Ctrl+Z to pause the process
4. Run bg to put the paused process in the background and resume it.
5. Run disown [pid] (process ID is optional, defaults to last process) to disown the process. To get a list of jobs simply type jobs before.
6. Exit the SSH session by running logout.

Usage of the disown command:

disown [-ar] [-h] [jobspec ... | pid ... ]
              Without  options,  remove  each jobspec from the table of active
              jobs.  If jobspec is not present, and neither the -a nor the  -r
              option  is  supplied, the current job is used.  If the -h option
              is given, each jobspec is not removed from  the  table,  but  is
              marked  so  that  SIGHUP  is  not  sent  to the job if the shell
              receives a SIGHUP.  If no jobspec is  supplied,  the  -a  option
              means  to  remove or mark all jobs; the -r option without a job‐
              spec argument restricts operation to running jobs.   The  return
              value is 0 unless a jobspec does not specify a valid job.