set statements don't appear to work in my batch file

I've found what appears to be an explanation of my problem here

DOS batch: Why are my set commands resulting in nothing getting stored?

but I don't really understand the explanation.

Here is my script...

for /R /d  %%f in (\Product\Database\SQL\Project\Model\Scripts\*) DO (

REM echo %%f
SET !LOAD_FILE_FILTER= %%f\*.sql
echo file: %!LOAD_FILE_FILTER%
CALL %!BATCH_FILE% -u %!USER_NAME% -p %!PASSWORD% -s %!SERVER_NAME% -d %!DATABASE_NAME% -f %!LOAD_FILE_FILTER% -o %!LOG_FILE%
IF %!EchoErrors%==1 (
    ECHO [
    TYPE %!LOG_FILE%
    ECHO ]
)

)

The echo always prints file: *.sql and the script I pass this var to always complains LOAD_FILE_FILTER is empty.

I have tried adding setlocal EnableDelayedExpansion as suggested in the article but it doesn't solve the problem. The echo file: %!LOAD_FILE_FILTER% always prints the last subdirectory in the directory I'm running from. The echo %%f always prints the correct value.

What does the '!' behind the variable do for/to me?

On a side note, could someone explain to me the difference between

SET !VAR and SET VAR

%VAR& &!VAR& !VAR! %%VAR

We are going to start with a simple case

set "var="
set "var=test" 
echo %var%

Reading the code, it removes the content of the variable, assigns it a new value and echoes it.

Let's change it a bit concatenating the last two commands

set "var="
set "var=test" & echo %var%

"Same" code, but in this case the output to console will not show the value in the variable.

Why? In batch files, lines to execute are parsed and then executed. During the parse phase, every variable read operation (where you retrieve the value of the variable) is replaced with the value stored inside the variable at parse time. Once this is done, the resulting command is executed. So, in the previous sample when the second line is parsed, it is converted to

set "var=test" & echo 

now, there are no read operations on the line and no value to echo, as when the line was readed the variable didn't hold any value (it will be assigned when the line is executed) so the read operation has been replaced with nothing. At this point, the code is executed and the perceived behaviour is that the set command failed as we don't get the "obvious" value echoed to console.

This behaviour is also found in blocks. A block is a set of lines enclosed in parenthesis (usually for and if constructs) and are handled by the parser as if all the lines in the block are only one line with concatenated commands. The full block is readed, all variable read operations removed and replaced with the value inside the variables, and then the full block, with no variable references inside is executed.

At execution time there are no read operation on variables inside the block, only its initial values, so, any value assigned to a variable inside the block can not be retrieved inside the same block, as there isn't any read operation.

So, in this code

set "test=before"
if defined test (
    set "test=after"
    echo %test%
)

after the first set is executed, the block (the if command and all the code enclosed in its parenthesis) will be parsed and converted into

if defined test (
    set "test=after"
    echo before
)

showing the "wrong" value.

The usual way to deal with it is to use delayed expansion. It will allow you to change, where needed, the syntax to read the variable from %var% into !var!, indicating to the parser that the read operation must not be removed at parse time, but delayed until the command is executed.

setlocal enabledelayedexpansion
set "var="
set "var=test" & echo !var!

The now third line is converted at parse time to

set "var=test" & echo !var!

yes, the variable reference is not removed. The read operation is delayed until the echo command will be executed, when the value of the variable has been changed.

So

%var% is a variable reference that will be replaced at parse time

!var! is a variable reference that will be replaced at execution time

%x with x a single character is usually a for replaceable parameter, a variable that will hold the current element being interated. By its own nature, will be expanded at execution time. The syntax with a single percent sign is used at command line. Inside batch files the percent sign need to be escaped and the syntax to refer to the replaceable parameters is %%x