Why you don't read lines with "for" in bash?

You are talking about for line in $(cat file); do and similar constructs. This is:

Inefficient because bash has to spawn a subshell and execute cat, read cat's output – the entire file – into memory, parse it (which is the slowest part), only then iterate over all data
Unreliable because bash performs word-splitting on the data – not only does it split on newline characters, but also on anything in $IFS (spaces, tabs...)

(If you use $(<...) instead of $(cat ...), you save two milliseconds on Linux, but all other downsides remain.)

A much better option is to use read in a while loop:

while read -r line; do
    echo "Input: $line"
done < myfile.txt

Or from a program:

cat file1 file2 file3 |
while read -r line; do
    echo "Input: $line"
done

This only reads as much as is needed, does not perform unnecessary processing but allows custom field splitting, and is many times faster and less resource-demanding on large files.

If you're trying to work with the output of find, you should use the same pattern:

find . -name "foo" -type f -print0 | while read -r -d '' file; do
    echo "File: $file"
done

Why you don't read lines with "for" in bash?

Related

Recent Posts