Taking Out 'Access denied " Lines

When I use find to see all the pdf files in the /home directory, I am seeing access denied. To eliminate them I tried:

find /home -iname "*.pdf" | grep -v "access denied"

However, the result is the same. How can I get rid of these lines?

Solution 1:

What you tried didn't work because the access denied output are errors and sent on STDERR instead of STDOUT which is piped to grep.

You can avoid seeing those errors by redirecting only STDERR

find /home -iname "*.pdf" 2>/dev/null

Or as David Foerster commented we can more succinctly close STDERR

find /home -iname "*.pdf" 2>&-

However, I suspect you actually only want to search your home rather than other users', so perhaps you really want

find ~ -iname "*.pdf"

If that throws errors there may be some wrong ownerships in your local config, which you should investigate.

Solution 2:

The access denied is probably being printed on stderr rather than stdout.

Try this:

find /home -iname "*.pdf" 2>&1 | grep -v "access denied"

The 2>&1 redirects the output from stderr to stdout, so that grep -v can do its job. (By default, | only pipes stdout and not stderr.)

Solution 3:

You probably mean "Permission denied"—which is what find in Ubuntu shows you when you cannot access something because of file permissions—rather than "access denied".

One fully general command that does this correctly (and, as a bonus, is portable to other *nixes, so long as the error message is the same) is:

(find 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-

(Usually you want to pass some arguments to find. Those go before the first redirection 3>&1.)

However, often you will able to use something simpler. For example, you can probably use process substitution. Details follow.

The Most Common Methods and Their Limitations

The two typical approaches are to throw away stderr (as in Zanna's answer) or to redirect stderr to stdout and filter stdout (as in Android Dev's answer). Although they have the advantage of being simple to write and are often reasonable choices, these approaches are not ideal.

Discarding everything sent to stderr—such as by redirecting it to the null device with 2>/dev/null or by closing it with 2>&-—runs the risk of missing errors other than "Permission denied".

"Permission denied" is probably the most common error seen when running find, but it is far from the only possible error, and if another does occur, you may want to know about it. In particular, find reports "No such file or directory" if a starting point does not exist. With multiple starting points, find may still return some useful results and appear to work. For example, if a and c exist but b doesn't, find a b c -name x prints results in a, then "No such file or directory" for b, then results in c.

Combining stdout and stderr together into stdout and piping it to grep or some other command to filter it—as with 2>&1 | grep ... or |& grep ...—runs the risk of unintentionally filtering out a file whose name contains the message being filtered.

For example, if you filter out lines that contain "Permission denied" then you will also drop search results showing filenames like "Permission denied messages.txt". This would probably happen by accident, though it would also be possible for a file to be given a specially crafted name to thwart your searches.

Filtering the combined streams has another problem, which cannot be mitigated by filtering more selectively (such as with grep -vx 'find: .*: Permission denied' on the right side of the pipe). Some find actions, including the -print action that is implicit when you specify no action, determine how to output filenames based on whether or not stdout is a terminal.

• If it isn't a terminal, then the filenames are output as-is even if they contain weird characters like newlines and control characters that could change the behavior of your terminal. If it is a terminal, then these characters are suppressed and ? is printed instead.
• This is usually what you want. If you are going to process filenames further, they must be output literally. However, if you are going to display them, a filename with a newline could otherwise mimic multiple filenames, and a filename with a sequence of backspace characters could appear to be a different name. Other problems are also possible, such as filenames containing escape sequences that change the colors in your terminal.
• But piping the search results through another command (like grep) causes find no longer to see a terminal. (More precisely, it causes its stdout not to be a terminal.) Then weird characters are output literally. But if all the command on the right side of the pipe does is (a) remove lines that look like "Permission denied" messages and (b) print what's left, then you're still subject to the sort of shenanigans that find's terminal detection is intended to prevent.
• See the UNUSUAL FILENAMES section of man find for more information, including the behavior of each of the actions that print filenames. ("Many of the actions of find result in the printing of data which is under the control of other users...") See also sections 3.3.2.1, 3.3.2.2, and 3.3.2.3 of the GNU Findutils reference manual.

The above discussion of unusual filenames pertains to GNU find, which is the find implementation in GNU/Linux systems including Ubuntu.

Leaving Standard Output Alone While Filtering Standard Error

What you really want here is to leave stdout intact while piping stderr to grep. Unfortunately there is no simple syntax for this. | pipes stdout, and some shells (including bash) support |& to pipe both streams—or you can redirect stderr to stdout first with 2>&1 |, which has the same effect. But the commonly used shells don't provide a syntax to pipe stderr only.

You can still do this. It's just awkward. One way is to swap stdout with stderr, so that search results are on stderr and errors are on stdout, then pipe stdout to grep for filtering:

find args 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied'

Usually you'll pass arguments to find, such as starting points (the places to search from, which are usually directories) and predicates (tests and actions). These go in place of args above.

This works by introducing a new file descriptor to hold onto one of the two standard streams you want to swap, performing redirections to swap them, and closing the new file descriptor.

• File descriptor 1 is stdout and 2 is stderr (and the unredirected 0 is stdin). But you can also redirect using other file descriptors. This can be used to open, or keep open, a file or device.
• 3>&1 redirects file descriptor 3 to stdout, so that when stdout (file descriptor 1) is subsequently redirected, the original stdout can still be written to easily.
• 1>&2 redirects stdout to stderr. Since file descriptor 3 is still the original stdout, it can still be accessed.
• 2>&3 redirects stderr to file descriptor 3, which is the original stdout.
• 3>&- closes file descriptor 3, which is no longer needed.
• For more information, see How to pipe stderr, and not stdout? and IO Redirection - Swapping stdout and stderr (Advanced) and especially pipe only stderr through a filter.

However, this method has the disadvantage that search results are sent to stderr and errors are sent to stdout. If you're running this command directly in an interactive shell and not piping or redirecting the output any further, then that doesn't really matter. Otherwise, it may be a problem. If you put that command in a script, and then someone (perhaps you, later) redirects or pipes its output, it does not behave as expected.

The solution is to swap the streams back after you're done filtering the output. Applying the same redirections shown above on the right side of the pipeline won't achieve this, because | only pipes stdout, so that side of the pipeline only receives output that was originally sent to stderr (because the streams were swapped) and not the original stdout output. Instead, you can use ( ) to run the above command in a subshell (related), then apply the swapping redirections to that:

(find args 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-

It's the grouping, not specifically the subshell, that makes this work. If you prefer, you can use { ;}:

{ find args 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied'; } 3>&1 1>&2 2>&3 3>&-

A Less Cumbersome Way: Process Substitution

Some shells, including Bash on systems that can support it (including GNU/Linux systems like Ubuntu), let you perform process substitution, which allows you to run a command and redirect to/from one of its streams. You can redirect the find command's stderr to a grep command that filters it, and redirect that grep command's stdout to stderr.

find args 2> >(grep -Fv 'Permission denied' >&2)

Credit goes to to Android Dev for this idea.

• See also Pinko's answer to How to pipe stderr, and not stdout?
• Both those examples use 1>&2. I have used >&2, which is equivalent (related). Use whichever you like.

Although bash supports process substitution, sh in Ubuntu is dash, which doesn't. It will give you "Syntax error: redirection unexpected" if you try to use this method, whereas the method of swapping stdout and stderr will still work. Furthermore, when bash runs in POSIX mode, support for process substitution is turned off.

One situation where bash runs in POSIX mode is when it is invoked as sh¹. Therefore, on an OS like Fedora where bash provides /bin/sh, or if you have made the /bin/sh symlink point to bash yourself on Ubuntu, process substitution still does not work in an sh script, without a prior command to turn POSIX mode off. Your best bet, if you want to use this method in a script, is to put #!/bin/bash at the top instead of #!/bin/sh, if you aren't already.

^{1: In this situation, bash turns on POSIX mode automatically after it runs the commands in its startup scripts.}

An Example

It is useful to be able to test these commands. To do this, I create a tmp subdirectory of the current directory and populate it with some files and directories, taking permissions away from one of them to trigger a "Permission denied" error in find.

mkdir tmp; cd tmp; mkdir a b c; touch w a/x 'a/Permission denied messages.txt' b/y c/z; chmod 0 b

One of the directories that is accessible includes a file with "Permission denied" in its name. Running find with no redirections or pipes shows this file, but also shows the actual "Permission denied" error for another directory that isn't accessible:

ek@Io:~/tmp$ find
.
./a
./a/Permission denied messages.txt
./a/x
./c
./c/z
./w
./b
find: ‘./b’: Permission denied

Piping both stdout and stderr to grep and filtering out lines that contain "Permission denied" makes the error message go away but also hides the search result for the file with that phrase in its name:

ek@Io:~/tmp$ find |& grep -Fv 'Permission denied'
.
./a
./a/x
./c
./c/z
./w
./b

find 2>&1 | grep -Fv 'Permission denied' is equivalent and produces the same output.

The methods shown above for filtering out "Permission denied" only from error messages—and not from search results—are successful. For example, here's the method where stdout and stderr are swapped:

ek@Io:~/tmp$ (find 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-
.
./a
./a/Permission denied messages.txt
./a/x
./c
./c/z
./w
./b

find args 2> >(grep -Fv 'Permission denied' >&2) produces the same output.

You can trigger a different error message to ensure that lines sent to stderr that don't contain the text "Permission denied" are still allowed through. For example, here I have run find with the current directory (.) as one starting point, but the nonexistent directory foo as another:

ek@Io:~/tmp$ (find . foo 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-
.
./a
./a/Permission denied messages.txt
./a/x
./c
./c/z
./w
./b
find: ‘foo’: No such file or directory

Checking That find's Standard Output is Still a Terminal

We can also see which commands cause special characters, such as newlines, to be displayed literally. (This can be done separately from the demonstration above, and it doesn't need to be in the tmp directory.)

Make a file with a newline in its name:

touch $'abc\ndef'

Usually we use directories as starting points for find, but files work too:

$ find abc*
abc?def

Piping stdout to another command causes the newline to be outputted literally, creating the false impression of two separate search results abc and def. We can test that with cat:

$ find abc* | cat
abc
def

Redirecting just stderr does not cause this problem:

$ find abc* 2>/dev/null
abc?def

Nor does closing it:

$ find abc* 2>&-
abc?def

Piping to grep does cause the problem:

$ find abc* |& grep -Fv 'Permission denied'
abc
def

(Replacing |& with 2>&1 | is equivalent and produces the same output.)

Swapping stdout and stderr and piping stdout does not cause the problem—find's stdout becomes stderr, which is not piped:

$ find abc* 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied'
abc?def

Grouping that command and swapping the streams back does not cause the problem:

$ (find abc* 3>&1 1>&2 2>&3 3>&- | grep -Fv 'Permission denied') 3>&1 1>&2 2>&3 3>&-
abc?def

(The { ;} version produces the same output.)

Using process substitution to filter stderr does not cause the problem either:

$ find abc* 2> >(grep -Fv 'Permission denied' >&2)
abc?def