How do file descriptors work?
Can someone tell me why this does not work? I'm playing around with file descriptors, but feel a little lost.
#!/bin/bash
echo "This"
echo "is" >&2
echo "a" >&3
echo "test." >&4
The first three lines run fine, but the last two error out. Why?
Solution 1:
File descriptors 0, 1 and 2 are for stdin, stdout and stderr respectively.
File descriptors 3, 4, .. 9 are for additional files. In order to use them, you need to open them first. For example:
exec 3<> /tmp/foo #open fd 3.
echo "test" >&3
exec 3>&- #close fd 3.
For more information take a look at Advanced Bash-Scripting Guide: Chapter 20. I/O Redirection.
Solution 2:
It's an old question but one thing needs clarification.
While the answers by Carl Norum and dogbane are correct, the assumption is to change your script to make it work.
What I'd like to point out is that you don't need to change the script:
#!/bin/bash
echo "This"
echo "is" >&2
echo "a" >&3
echo "test." >&4
It works if you invoke it differently:
./fdtest 3>&1 4>&1
which means to redirect file descriptors 3 and 4 to 1 (which is standard output).
The point is that the script is perfectly fine in wanting to write to descriptors other than just 1 and 2 (stdout and stderr) if those descriptors are provided by the parent process.
Your example is actually quite interesting because this script can write to 4 different files:
./fdtest >file1.txt 2>file2.txt 3>file3.txt 4>file4.txt
Now you have the output in 4 separate files:
$ for f in file*; do echo $f:; cat $f; done
file1.txt:
This
file2.txt:
is
file3.txt:
a
file4.txt:
test.
What is more interesting about it is that your program doesn't have to have write permissions for those files, because it doesn't actually open them.
For example, when I run sudo -s to change user to root, create a directory as root, and try to run the following command as my regular user (rsp in my case) like this:
# su rsp -c '../fdtest >file1.txt 2>file2.txt 3>file3.txt 4>file4.txt'
I get an error:
bash: file1.txt: Permission denied
But if I do the redirection outside of su:
# su rsp -c '../fdtest' >file1.txt 2>file2.txt 3>file3.txt 4>file4.txt
(note the difference in single quotes) it works and I get:
# ls -alp
total 56
drwxr-xr-x 2 root root 4096 Jun 23 15:05 ./
drwxrwxr-x 3 rsp rsp 4096 Jun 23 15:01 ../
-rw-r--r-- 1 root root 5 Jun 23 15:05 file1.txt
-rw-r--r-- 1 root root 39 Jun 23 15:05 file2.txt
-rw-r--r-- 1 root root 2 Jun 23 15:05 file3.txt
-rw-r--r-- 1 root root 6 Jun 23 15:05 file4.txt
which are 4 files owned by root in a directory owned by root - even though the script didn't have permissions to create those files.
Another example would be using chroot jail or a container and run a program inside where it wouldn't have access to those files even if it was run as root and still redirect those descriptors externally where you need, without actually giving access to the entire file system or anything else to this script.
The point is that you have discovered a very interesting and useful mechanism. You don't have to open all the files inside of your script as was suggested in other answers. Sometimes it is useful to redirect them during the script invocation.
To sum it up, this:
echo "This"
is actually equivalent to:
echo "This" >&1
and running the program as:
./program >file.txt
is the same as:
./program 1>file.txt
The number 1 is just a default number and it is stdout.
But even this program:
#!/bin/bash
echo "This"
can produce a "Bad descriptor" error. How? When run as:
./fdtest2 >&-
The output will be:
./fdtest2: line 2: echo: write error: Bad file descriptor
Adding >&- (which is the same as 1>&-) means closing the standard output. Adding 2>&- would mean closing the stderr.
You can even do a more complicated thing. Your original script:
#!/bin/bash
echo "This"
echo "is" >&2
echo "a" >&3
echo "test." >&4
when run with just:
./fdtest
prints:
This
is
./fdtest: line 4: 3: Bad file descriptor
./fdtest: line 5: 4: Bad file descriptor
But you can make descriptors 3 and 4 work, but number 1 fail by running:
./fdtest 3>&1 4>&1 1>&-
It outputs:
./fdtest: line 2: echo: write error: Bad file descriptor
is
a
test.
If you want descriptors both 1 and 2 fail, run it like this:
./fdtest 3>&1 4>&1 1>&- 2>&-
You get:
a
test.
Why? Didn't anything fail? It did but with no stderr (file descriptor number 2) you didn't see the error messages!
I think it's very useful to experiment this way to get a feeling of how the descriptors and their redirection work.
Your script is a very interesting example indeed - and I argue that it is not broken at all, you were just using it wrong! :)
Solution 3:
It's failing because those file descriptors don't point to anything! The normal default file descriptors are the standard input 0, the standard output 1, and the standard error stream 2. Since your script isn't opening any other files, there are no other valid file descriptors. You can open a file in bash using exec. Here's a modification of your example:
#!/bin/bash
exec 3> out1 # open file 'out1' for writing, assign to fd 3
exec 4> out2 # open file 'out2' for writing, assign to fd 4
echo "This" # output to fd 1 (stdout)
echo "is" >&2 # output to fd 2 (stderr)
echo "a" >&3 # output to fd 3
echo "test." >&4 # output to fd 4
And now we'll run it:
$ ls
script
$ ./script
This
is
$ ls
out1 out2 script
$ cat out*
a
test.
$
As you can see, the extra output was sent to the requested files.
Solution 4:
To add on to the answer from rsp and respond the question in the comments of that answer from @MattClimbs.
You can test if the file descriptor is open or not by attempting to redirect to it early and if it fails, open the desired numbered file descriptor to something like /dev/null. I do this regularly within scripts and leverage the additional file descriptors to pass back additional details or responses beyond return #.
script.sh
#!/bin/bash
2>/dev/null >&3 || exec 3>/dev/null
2>/dev/null >&4 || exec 4>/dev/null
echo "This"
echo "is" >&2
echo "a" >&3
echo "test." >&4
The stderr is redirected to /dev/null to discard the possible bash: #: Bad file descriptor response and the || is used to process the following command exec #>/dev/null when the previous one exits with a non zero status. In the event that the file descriptor is already opened, the two tests would return a zero status and the exec ... command would not be executed.
Calling the script without any redirections yields:
# ./script.sh
This
is
In this case, the redirections for a and test are shipped off to /dev/null
Calling the script with a redirection defined yields:
# ./script.sh 3>temp.txt 4>>temp.txt
This
is
# cat temp.txt
a
test.
The first redirection 3>temp.txt overwrites the file temp.txt while 4>>temp.txt appends to the file.
In the end, you can define default files to redirect to within the script if you want something other than /dev/null or you can change the execution method of the script and redirect those extra file descriptors anywhere you want.