Iterate over lines instead of words in a for loop of shell script

Solution 1:

The for loop is not designed to loop over "lines". Instead it loops over "words", or "fields".

The idiomatic way to loop over lines is to use a while loop in combination with read.

ioscan -m dsf | while read -r line
do
  printf '%s\n' "$line"
done

Note that the while loop is in a subshell because of the pipe. This can cause some consfusion with variable scope. In bash you can work around this by using process substitution.

while read -r line
do
  printf '%s\n' "$line"
done < <(ioscan -m dsf)

see also http://mywiki.wooledge.org/BashFAQ/024

The for loop splits the values to loop over using the characters in the $IFS (Internal field separator) variable as separators. Usually $IFS contains a space, a tab, and a newline. That means the for loop will loop over the "words", not over the lines.

If you insist on using a for loop to loop over lines you have to change the value of $IFS to only newline. But if you do this you have to save the old value of $IFS and restore that after the loop, because many other things also depend on $IFS.

OLDIFS="$IFS"
IFS=$'\n' # bash specific
for line in $(ioscan -m dsf)
do
  printf '%s\n' "$line"
done
IFS="$OLDIFS"

in POSIX shells, that have no ANSI-C Quoting ($'\n'), you can do it like this:

IFS='
'

that is: put an actual new line between the quotes.

Alternatively you can use a subshell to contain the change to $IFS:

(
  # changes to variables in the subshell stay in the subshell
  IFS=$'\n'
  for line in $(ioscan -m dsf)
  do
    printf '%s\n' "$line"
  done
)
# $IFS is not changed outside of the subshell

But beware the command in the loop may itself depends on some sane setting for $IFS. Then you have to restore the $IFS before executing the command and set again before the next loop or some such. I do not recommend messing with $IFS. Too many commands depend on some sane values in $IFS and changing it is an endless nightmare of obscure bug hunting.

Solution 2:

Using for

for l in $() performs word splitting based on IFS:

$ for l in $(printf %b 'a b\nc'); do echo "$l"; done
a
b
c
$ IFS=$'\n'; for l in $(printf %b 'a b\nc'); do echo "$l"; done
a b
c

IFS doesn't have to be set back if it is not used later.

for l in $() also performs pathname expansion:

$ printf %b 'a\n*\n' > file.txt
$ IFS=$'\n'
$ for l in $(<file.txt); do echo "$l"; done
a
file.txt
$ set -f; for l in $(<file.txt); do echo "$l"; done; set +f
a
*

If IFS=$'\n', linefeeds are stripped and collapsed:

$ printf %b '\n\na\n\nb\n\n' > file.txt
$ IFS=$'\n'; for l in $(<file.txt); do echo "$l"; done
a
b

$(cat file.txt) (or $(<file.txt)) also reads the whole file to memory.

Using read

Without -r backslashes are used for line continuation and removed before other characters:

$ cat file.txt
\1\\2\
3
$ cat file.txt | while read l; do echo "$l"; done
1\23
$ cat file.txt | while read -r l; do echo "$l"; done
\1\\2\
3

Characters in IFS are stripped from the start and end of lines but not collapsed:

$ printf %b '1  2 \n\t3\n' | while read -r l; do echo "$l"; done
1  2
3
$ printf %b ' 1  2 \n\t3\n' | while IFS= read -r l; do echo "$l"; done
 1  2 
    3

If the last line doesn't end with a newline, read assigns l to it but exits before the body of the loop:

$ printf 'x\ny' | while read l; do echo $l; done
x
$ printf 'x\ny' | while read l || [[ $l ]]; do echo $l; done
x
y

If a while loop is in a pipeline, it is also in a subshell, so variables are not visible outside it:

$ x=0; seq 3 | while read l; do let x+=l; done; echo $x
0
$ x=0; while read l; do let x+=l; done < <(seq 3); echo $x
6
$ x=0; x=8 | x=9; echo $x
0

Iterate over lines instead of words in a for loop of shell script

Solution 1:

Solution 2:

Using for

Using read

Related

Recent Posts