Run background (async) cmd with sync output
Solution 1:
With GNU parallel
. Basic example:
parallel -j 40 --group --keep-order curl ::: "${urls[@]}"
echo 'stuff'
-j 40
means we assign 40 jobslots, i.e. we allow up to 40 parallel jobs (adjust it to your needs and abilities). If you supply more URLs then the 41st one will be processed after some slot gets available. All URLs will be processed but at any moment there will be at most 40 jobs running in parallel.
Other options used:
--group
Group output. Output from each job is grouped together and is only printed when the command is finished. Stdout (standard output) first followed by stderr (standard error). […]
(source)
which is the default, so usually you don't have to use it explicitly.
--keep-order
-k
Keep sequence of output same as the order of input. Normally the output of a job will be printed as soon as the job completes. […]-k
only affects the order in which the output is printed - not the order in which jobs are run.
(source)
Notes:
-
In my example
parallel
is not in the background and is run synchronously (soecho
runs after it); stillcurl
s run in parallel, asynchronously. -
In Debian GNU
parallel
is in a package namedparallel
. Basic variant of the tool (frommoreutils
, at least in Debian) is less powerful. -
parallel
is an external command. If the array is large enough then withparallel … ::: "${urls[@]}"
you will hitargument list too long
. Use this instead:printf '%s\n' "${urls[@]}" | parallel …
It will work because in Bash
printf
is a builtin and therefore everything before|
is handled internally by Bash. -
${urls[@]}
is properly double-quoted (in your code${urls[@]}
and$i
are unquoted, this is wrong).
GNU parallel
can call exported Bash functions. This allows us to solve what you called the actual problem:
getvideo() {
curl "$1" > "${1##*/}"
}
export -f getvideo
urls=(
'https://www.example1.com/stccdtu.mp4'
'https://www.example2.com/dyubdf.mp4'
)
parallel -j 40 --group --keep-order getvideo ::: "${urls[@]}"
echo 'stuff'
If you don't know what ${1##*/}
does, read this another answer of mine.