Bash: Capture output of command run in background
I'm trying to write a bash script that will get the output of a command that runs in the background. Unfortunately I can't get it to work, the variable I assign the output to is empty - if I replace the assignment with an echo command everything works as expected though.
#!/bin/bash
function test {
echo "$1"
}
echo $(test "echo") &
wait
a=$(test "assignment") &
wait
echo $a
echo done
This code produces the output:
echo
done
Changing the assignment to
a=`echo $(test "assignment") &`
works, but it seems like there should be a better way of doing this.
Bash has indeed a feature called Process Substitution to accomplish this.
$ echo <(yes)
/dev/fd/63
Here, the expression <(yes) is replaced with a pathname of a (pseudo device) file that is connected to the standard output of an asynchronous job yes (which prints the string y in an endless loop).
Now let's try to read from it:
$ cat /dev/fd/63
cat: /dev/fd/63: No such file or directory
The problem here is that the yes process terminated in the meantime because it received a SIGPIPE (it had no readers on stdout).
The solution is the following construct
$ exec 3< <(yes) # Save stdout of the 'yes' job as (input) fd 3.
This opens the file as input fd 3 before the background job is started.
You can now read from the background job whenever you prefer. For a stupid example
$ for i in 1 2 3; do read <&3 line; echo "$line"; done
y
y
y
Note that this has slightly different semantics than having the background job write to a drive backed file: the background job will be blocked when the buffer is full (you empty the buffer by reading from the fd). By contrast, writing to a drive-backed file is only blocking when the hard drive doesn't respond.
Process substitution is not a POSIX sh feature.
Here's a quick hack to give an asynchronous job drive backing (almost) without assigning a filename to it:
$ yes > backingfile & # Start job in background writing to a new file. Do also look at `mktemp(3)` and the `sh` option `set -o noclobber`
$ exec 3< backingfile # open the file for reading in the current shell, as fd 3
$ rm backingfile # remove the file. It will disappear from the filesystem, but there is still a reader and a writer attached to it which both can use it.
$ for i in 1 2 3; do read <&3 line; echo "$line"; done
y
y
y
Linux also recently got added the O_TEMPFILE option, which makes such hacks possible without the file ever being visible at all. I don't know if bash already supports it.
UPDATE:
@rthur, if you want to capture the whole output from fd 3, then use
output=$(cat <&3)
But note that you can't capture binary data in general: It's only a defined operation if the output is text in the POSIX sense. The implementations I know simply filter out all NUL bytes. Furthermore POSIX specifies that all trailing newlines must be removed.
(Please note also that capturing the output will result in OOM if the writer never stops (yes never stops). But naturally that problem holds even for read if the line separator is never written additionally)
One very robust way to deal with coprocesses in Bash is to use... the coproc builtin.
Suppose you have a script or function called banana you wish to run in background, capture all its output while doing some stuff and wait until it's done. I'll do the simulation with this:
banana() {
for i in {1..4}; do
echo "gorilla eats banana $i"
sleep 1
done
echo "gorilla says thank you for the delicious bananas"
}
stuff() {
echo "I'm doing this stuff"
sleep 1
echo "I'm doing that stuff"
sleep 1
echo "I'm done doing my stuff."
}
You will then run banana with the coproc as so:
coproc bananafd { banana; }
this is like running banana & but with the following extras: it creates two file descriptors that are in the array bananafd (at index 0 for output and index 1 for input). You'll capture the output of banana with the read builtin:
IFS= read -r -d '' -u "${bananafd[0]}" banana_output
Try it:
#!/bin/bash
banana() {
for i in {1..4}; do
echo "gorilla eats banana $i"
sleep 1
done
echo "gorilla says thank you for the delicious bananas"
}
stuff() {
echo "I'm doing this stuff"
sleep 1
echo "I'm doing that stuff"
sleep 1
echo "I'm done doing my stuff."
}
coproc bananafd { banana; }
stuff
IFS= read -r -d '' -u "${bananafd[0]}" banana_output
echo "$banana_output"
Caveat: you must be done with stuff before banana ends! if the gorilla is quicker than you:
#!/bin/bash
banana() {
for i in {1..4}; do
echo "gorilla eats banana $i"
done
echo "gorilla says thank you for the delicious bananas"
}
stuff() {
echo "I'm doing this stuff"
sleep 1
echo "I'm doing that stuff"
sleep 1
echo "I'm done doing my stuff."
}
coproc bananafd { banana; }
stuff
IFS= read -r -d '' -u "${bananafd[0]}" banana_output
echo "$banana_output"
In this case, you'll obtain an error like this one:
./banana: line 22: read: : invalid file descriptor specification
You can check whether it's too late (i.e., whether you've taken too long doing your stuff) because after the coproc is done, bash removes the values in the array bananafd, and that's why we obtained the previous error.
#!/bin/bash
banana() {
for i in {1..4}; do
echo "gorilla eats banana $i"
done
echo "gorilla says thank you for the delicious bananas"
}
stuff() {
echo "I'm doing this stuff"
sleep 1
echo "I'm doing that stuff"
sleep 1
echo "I'm done doing my stuff."
}
coproc bananafd { banana; }
stuff
if [[ -n ${bananafd[@]} ]]; then
IFS= read -r -d '' -u "${bananafd[0]}" banana_output
echo "$banana_output"
else
echo "oh no, I took too long doing my stuff..."
fi
Finally, if you really don't want to miss any of gorilla's moves, even if you take too long for your stuff, you could copy banana's file descriptor to another fd, 3 for example, do your stuff and then read from 3:
#!/bin/bash
banana() {
for i in {1..4}; do
echo "gorilla eats banana $i"
sleep 1
done
echo "gorilla says thank you for the delicious bananas"
}
stuff() {
echo "I'm doing this stuff"
sleep 1
echo "I'm doing that stuff"
sleep 1
echo "I'm done doing my stuff."
}
coproc bananafd { banana; }
# Copy file descriptor banana[0] to 3
exec 3>&${bananafd[0]}
stuff
IFS= read -d '' -u 3 output
echo "$output"
This will work very well! the last read will also play the role of wait, so that output will contain the complete output of banana.
That was great: no temp files to deal with (bash handles everything silently) and 100% pure bash!
Hope this helps!