How to run commands as in a queue

I need to do a lot of copying of various files to various folders. I can add all my copy commands to a bash script and then run that, but then I must wait until it finishes if I want to add more commands to that copying "queue".

Is there a way I can run commands as a queue and sort of add more commands to that queue while things are running?

Explained in a different way, I want to start a long running task. While that is running I want to start another one that does not actually start until the first one is done. And then add another after that last one and so on. Is this possible somehow?

Solution 1:

Append & to the end of your command to send it to the background, and then wait on it before running the next. For example:

$ command1 &
$ wait; command2 &
$ wait; command3 &
$ ...

Solution 2:

The at utility, best known for running commands at a specified time, also has a queue feature and can be asked to start running commands now. It reads the command to run from standard input.

echo 'command1 --option arg1 arg2' | at -q myqueue now
echo 'command2 ...' | at -q myqueue now

The batch command is equivalent to at -q b -m now (-m meaning the command output, if any, will be mailed to you, like cron does). Not all unix variants support queue names (-q myqueue); you may be limited to a single queue called b. Linux's at is limited to single-letter queue names.

Solution 3:

Both Brad and Mankoff's solutions are good suggestions. Another that's similar to a combination of both them would be to use GNU Screen to implement your queue. This has the advantage of being able to run in the background, you can check on it whenever, and queueing up new commands just pastes them into the buffer to be executed after the previous commands exit.

First, run:

$ screen -d -m -S queue

(incidentally, now's a good time to play with some awesome .screenrc files)

That will spawn up a background screen session for you named queue.

Now, queue up as many commands as you like:

screen -S queue -X stuff "echo first; sleep 4; echo second^M"

I'm doing multiple commands in the above just for testing. Your use case would probably look more like:

screen -S queue -X stuff "echo first^M"
screen -S queue -X stuff "echo second^M"

Note that the "^M" in my line above is a way to get an embedded newline that will be interpreted later after screen stuffs it into your existing bash shell. Use "CTL-V, " to get that sequence.

It'd be pretty easy to make some simple shell-scripts to automate that and queue up commands. Then, whenever you want to check the status of your background queue, you re-attach via:

screen -S queue -r

Technically, you don't even need to name your screen session and it will work fine, but once you get hooked on it, you're going to want to leave one running all the time anyway. ;-)

Of course, if you do that, another good way to do it would be to name one of the current windows "queue" and use:

screen -S queue -p queue -X stuff "command"

Solution 4:

There is a utility that I have used with great success for exactly the use case that you are describing. Recently I moved my primary laptop to new hardware, which involved moving some files to a NAS and the rest of them to the new machine.

This is how I did it.

Set up all machines involved with a network connection so that they can reach each other.
On the machine where you are moving files from (hereafter called source machine), install rsync with apt-get install rsync and the secret-sauce Task Spooler (website http://vicerveza.homeunix.net/~viric/soft/ts/). If you are on Debian the package name for ts is task-spooler and the executable is renamed to tsp to avoid name clash with the ts executable from the moreutils package. The Debian package linked from the website is perfectly installable on Ubuntu with dpkg -i task-spooler_0.7.3-1_amd64.deb or similar.
Also make sure that all machines have SSH installed with apt-get install openssh-server. On the source machine you need to set up SSH to allow passwordless login to the target machines. The method most will use is public-key authentication with ssh-agent (see https://www.google.se/search?q=ssh+public+key+authentication for examples of that), but I sometimes use an easier method which works just as well with password authentication. Add the following to your SSH client configuration (either ~/.ssh/config or /etc/ssh/ssh_config):
```
Host *
     ControlMaster auto
     ControlPath ~/.ssh/master-%r@%h:%p
```
Then open one terminal on the source machine, login with ssh target1, authenticate as usual, and then leave this terminal open. Notice that there is a socket file named ~/.ssh/master-user@target1:22, this is the file that will keep an authenticated master session open and allow subsequent passwordless connections for user (as long as the connection uses the same target hostname and port).

At this point you need to verify that you can login without being prompted for authentication to the target machines.
Now run ts rsync -ave ssh bigfile user@target1: for a single file or ts rsync -ave ssh bigdir user@target1: for a directory. With rsync it is important not to include a trailing slash on the directory (bigdir compared to bigdir/) or rsync will assume that you meant the equivalent of bigdir/* in most other tools.

Task Spooler will return the prompt and let you queue many of these commands in succession. Inspect the run queue with ts without arguments.

Task Spooler has many features such as rearranging the run queue, run a specific job only if another job ran successfully, etc. View the help with ts -h. I sometimes inspect the command output while it is running with ts -c.

There are other methods of doing this but for your use case they all include Task Spooler. I choose to use rsync over SSH to preserve file timestamps, which copying over SMB would not have.

Solution 5:

I need stuff like this fairly frequently too. I wrote a little utility called after that executes a command whenever some other process has finished. It looks like this:

#!/usr/bin/perl

my $pid = shift;
die "Usage: $0 <pid> <command...>" unless $pid =~ /^\d+$/ && @ARGV;

print STDERR "Queueing process $$ after process $pid\n";
sleep 1 while -e "/proc/$pid";
exec @ARGV;

You then run it like so:

% command1 arg1 arg2 ...  # creates pid=2853
% after 2853 command2 arg1 arg2 ... # creates pid=9564
% after 9564 command3 arg1 arg2 ...

The big advantage to this over some other approaches is that the first job doesn't need to be run in any special way, you can follow any process with your new job.