Getting ssh to execute a command in the background on target machine

This is a follow-on question to the How do you use ssh in a shell script? question. If I want to execute a command on the remote machine that runs in the background on that machine, how do I get the ssh command to return? When I try to just include the ampersand (&) at the end of the command it just hangs. The exact form of the command looks like this:

ssh user@target "cd /some/directory; program-to-execute &"

Any ideas? One thing to note is that logins to the target machine always produce a text banner and I have SSH keys set up so no password is required.

Solution 1:

I had this problem in a program I wrote a year ago -- turns out the answer is rather complicated. You'll need to use nohup as well as output redirection, as explained in the wikipedia artcle on nohup, copied here for your convenience.

Nohuping backgrounded jobs is for example useful when logged in via SSH, since backgrounded jobs can cause the shell to hang on logout due to a race condition [2]. This problem can also be overcome by redirecting all three I/O streams:
nohup myprogram > foo.out 2> foo.err < /dev/null &

Solution 2:

This has been the cleanest way to do it for me:-

ssh -n -f user@host "sh -c 'cd /whereever; nohup ./whatever > /dev/null 2>&1 &'"

The only thing running after this is the actual command on the remote machine

Solution 3:

Redirect fd's

Output needs to be redirected with &>/dev/null which redirects both stderr and stdout to /dev/null and is a synonym of >/dev/null 2>/dev/null or >/dev/null 2>&1.

Parantheses

The best way is to use sh -c '( ( command ) & )' where command is anything.

ssh askapache 'sh -c "( ( nohup chown -R ask:ask /www/askapache.com &>/dev/null ) & )"'

Nohup Shell

You can also use nohup directly to launch the shell:

ssh askapache 'nohup sh -c "( ( chown -R ask:ask /www/askapache.com &>/dev/null ) & )"'

Nice Launch

Another trick is to use nice to launch the command/shell:

ssh askapache 'nice -n 19 sh -c "( ( nohup chown -R ask:ask /www/askapache.com &>/dev/null ) & )"'

Solution 4:

If you don't/can't keep the connection open you could use screen, if you have the rights to install it.

user@localhost $ screen -t remote-command
user@localhost $ ssh user@target # now inside of a screen session
user@remotehost $ cd /some/directory; program-to-execute &

To detach the screen session: ctrl-a d

To list screen sessions:

screen -ls

To reattach a session:

screen -d -r remote-command

Note that screen can also create multiple shells within each session. A similar effect can be achieved with tmux.

user@localhost $ tmux
user@localhost $ ssh user@target # now inside of a tmux session
user@remotehost $ cd /some/directory; program-to-execute &

To detach the tmux session: ctrl-b d