How to substitute shell variables in complex text files

I have several text files in which I have introduced shell variables ($VAR1 or $VAR2 for instance).

I would like to take those files (one by one) and save them in new files where all variables would have been replaced.

To do this, I used the following shell script (found on StackOverflow):

while read line
do
    eval echo "$line" >> destination.txt
done < "source.txt"

This works very well on very basic files.

But on more complex files, the "eval" command does too much:

• Lines starting with "#" are skipped

• XML files parsing results in tons of errors

Is there a better way to do it? (in shell script... I know this is easily done with Ant for instance)

Kind regards

Solution 1:

Looking, it turns out on my system there is an envsubst command which is part of the gettext-base package.

So, this makes it easy:

envsubst < "source.txt" > "destination.txt"

Note if you want to use the same file for both, you'll have to use something like moreutil's sponge, as suggested by Johnny Utahh: envsubst < "source.txt" | sponge "source.txt". (Because the shell redirect will otherwise empty the file before its read.)

Solution 2:

In reference to answer 2, when discussing envsubst, you asked:

How can I make it work with the variables that are declared in my .sh script?

The answer is you simply need to export your variables before calling envsubst.

You can also limit the variable strings you want to replace in the input using the envsubst SHELL_FORMAT argument (avoiding the unintended replacement of a string in the input with a common shell variable value - e.g. $HOME).

For instance:

export VAR1='somevalue' VAR2='someothervalue'
MYVARS='$VAR1:$VAR2'

envsubst "$MYVARS" <source.txt >destination.txt

Will replace all instances of $VAR1 and $VAR2 (and only VAR1 and VAR2) in source.txt with 'somevalue' and 'someothervalue' respectively.

Solution 3:

I know this topic is old, but I have a simpler working solution without export the variables. Can be a oneliner, but I prefer to split using \ on line end.

var1='myVar1'\
var2=2\
var3=${var1}\
envsubst '$var1,$var3' < "source.txt" > "destination.txt"

#         ^^^^^^^^^^^    ^^^^^^^^^^     ^^^^^^^^^^^^^^^
# define which to replace   input            output

The variables need to be defined to the same line as envsubst is to get considered as environment variables.

The '$var1,$var3' is optional to only replace the specified ones. Imagine an input file containing ${VARIABLE_USED_BY_JENKINS} which should not be replaced.

Solution 4:

1. Define your ENV variable

$ export MY_ENV_VAR=congratulation

2. Create template file (in.txt) with following content

$MY_ENV_VAR

You can also use all other ENV variables defined by your system like (in linux) $TERM, $SHELL, $HOME...

3. Run this command to raplace all env-variables in your in.txt file and to write the result to out.txt

$ envsubst "`printf '${%s} ' $(sh -c "env|cut -d'=' -f1")`" < in.txt > out.txt

4. Check the content of out.txt file

$ cat out.txt

and you should see "congratulation".