Bash script processing limited number of commands in parallel

Solution 1:

Use the wait built-in:

process1 &
process2 &
process3 &
process4 &
wait
process5 &
process6 &
process7 &
process8 &
wait

For the above example, 4 processes process1 ... process4 would be started in the background, and the shell would wait until those are completed before starting the next set.

From the GNU manual:

wait [jobspec or pid ...]

Wait until the child process specified by each process ID pid or job specification jobspec exits and return the exit status of the last command waited for. If a job spec is given, all processes in the job are waited for. If no arguments are given, all currently active child processes are waited for, and the return status is zero. If neither jobspec nor pid specifies an active child process of the shell, the return status is 127.

Solution 2:

See parallel. Its syntax is similar to xargs, but it runs the commands in parallel.

Solution 3:

In fact, xargs can run commands in parallel for you. There is a special -P max_procs command-line option for that. See man xargs.

Solution 4:

You can run 20 processes and use the command:

wait

Your script will wait and continue when all your background jobs are finished.