Why does delayed expansion fail when inside a piped block of code?

Here is a simple batch file that demonstrates how delayed expansion fails if it is within a block that is being piped. (The failure is toward the end of the script) Can anyone explain why this is?

I have a work-around, but it requires creation of a temporary file. I initially ran into this problem while working on Find files and sort by size in a Windows batch file

@echo off
setlocal enableDelayedExpansion

set test1=x
set test2=y
set test3=z

echo(

echo NORMAL EXPANSION TEST
echo Unsorted works
(
  echo %test3%
  echo %test1%
  echo %test2%
)
echo(
echo Sorted works
(
  echo %test3%
  echo %test1%
  echo %test2%
) | sort

echo(
echo ---------
echo(

echo DELAYED EXPANSION TEST
echo Unsorted works
(
  echo !test3!
  echo !test1!
  echo !test2!
)
echo(
echo Sorted fails
(
  echo !test3!
  echo !test1!
  echo !test2!
) | sort
echo(
echo Sort workaround
(
  echo !test3!
  echo !test1!
  echo !test2!
)>temp.txt
sort temp.txt
del temp.txt

Here are the results

NORMAL EXPANSION TEST
Unsorted works
z
x
y

Sorted works
x
y
z

---------

DELAYED EXPANSION TEST
Unsorted works
z
x
y

Sorted fails
!test1!
!test2!
!test3!

Sort workaround
x
y
z

As Aacini shows, it seems that many things fail within a pipe.

echo hello | set /p var=
echo here | call :function

But in reality it's only a problem to understand how the pipe works.

Each side of a pipe starts its own cmd.exe in its own ascynchronous thread.
That is the cause why so many things seem to be broken.

But with this knowledge you can avoid this and create new effects

echo one | ( set /p varX= & set varX )
set var1=var2
set var2=content of two
echo one | ( echo %%%var1%%% )
echo three | echo MYCMDLINE %%cmdcmdline%%
echo four  | (cmd /v:on /c  echo 4: !var2!)

Update 2019-08-15:
As discovered at Why does `findstr` with variable expansion in its search string return unexpected results when involved in a pipe?, cmd.exe is only used if the command is internal to cmd.exe, if the command is a batch file, or if the command is enclosed in a parenthesized block. External commands not enclosed within parentheses are launched in a new process without the aid of cmd.exe.

EDIT: In depth analysis

As dbenham shows, both sides of the pipes are equivalent for the expansion phases.
The main rules seems to be:

The normal batch parser phases are done
.. percent expansion
.. special character phase/block begin detection
.. delayed expansion (but only if delayed expansion is enabled AND it isn't a command block)

Start the cmd.exe with C:\Windows\system32\cmd.exe /S /D /c"<BATCH COMMAND>"
These expansions follows the rules of the cmd-line parser not the the batch-line parser.

.. percent expansion
.. delayed expansion (but only if delayed expansion is enabled)

The <BATCH COMMAND> will be modified if it's inside a parenthesis block.

(
echo one %%cmdcmdline%%
echo two
) | more

Called as C:\Windows\system32\cmd.exe /S /D /c" ( echo one %cmdcmdline% & echo two )", all newlines are changed to & operator.

Why the delayed expansion phase is affected by parenthesis?
I suppose, it can't expand in the batch-parser-phase, as a block can consist of many commands and the delayed expansion take effect when a line is executed.

(
set var=one
echo !var!
set var=two
) | more

Obviously the !var! can't be evaluated in the batch context, as the lines are executed only in the cmd-line context.

But why it can be evaluated in this case in the batch context?

echo !var! | more

In my opionion this is a "bug" or inconsitent behaviour, but it's not the first one

EDIT: Adding the LF trick

As dbenham shows, there seems to be some limitation through the cmd-behaviour that changes all line feeds into &.

(
  echo 7: part1
  rem This kills the entire block because the closing ) is remarked!
  echo part2
) | more

This results into
C:\Windows\system32\cmd.exe /S /D /c" ( echo 7: part1 & rem This ...& echo part2 ) "
The rem will remark the complete line tail, so even the closing bracket is missing then.

But you can solve this with embedding your own line feeds!

set LF=^


REM The two empty lines above are required
(
  echo 8: part1
  rem This works as it splits the commands %%LF%% echo part2  
) | more

This results to C:\Windows\system32\cmd.exe /S /D /c" ( echo 8: part1 %cmdcmdline% & rem This works as it splits the commands %LF% echo part2 )"

And as the %lf% is expanded while parsing the parenthises by the parser, the resulting code looks like

( echo 8: part1 & rem This works as it splits the commands 
  echo part2  )

This %LF% behaviour works always inside of parenthesis, also in a batch file.
But not on "normal" lines, there a single <linefeed> will stop the parsing for this line.

EDIT: Asynchronously is not the full truth

I said that the both threads are asynchronous, normally this is true.
But in reality the left thread can lock itself when the piped data isn't consumed by the right thread.
There seems to be a limit of ~1000 characters in the "pipe" buffer, then the thread is blocked until the data is consumed.

@echo off
(
    (
    for /L %%a in ( 1,1,60 ) DO (
            echo A long text can lock this thread
            echo Thread1 ##### %%a > con
        )
    )
    echo Thread1 ##### end > con
) | (
    for /L %%n in ( 1,1,6) DO @(
        ping -n 2 localhost > nul
        echo Thread2 ..... %%n
        set /p x=
    )
)

I wasn't sure if I should edit my question, or post this as an answer.

I already vaguely knew that a pipe executes both the left and the right side each in its own CMD.EXE "session". But Aacini's and jeb's responses forced me to really think about and investigate what is happening with pipes. (Thank you jeb for demonstrating what is happening when piping into SET /P!)

I developed this investigative script - it helps explain a lot, but also demonstrates some bizarre and unexpected behavior. I'll post the script, followed by the output. Finally I will provide some analysis.

@echo off
cls
setlocal disableDelayedExpansion
set var1=value1
set "var2="
setlocal enableDelayedExpansion

echo on
@echo NO PIPE - delayed expansion is ON
echo 1: %var1%, %var2%, !var1!, !var2!
(echo 2: %var1%, %var2%, !var1!, !var2!)

@echo(
@echo PIPE LEFT SIDE - Delayed expansion is ON
echo 1L: %%var1%%, %%var2%%, !var1!, !var2! | more
(echo 2L: %%var1%%, %%var2%%, !var1!, !var2!) | more
(setlocal enableDelayedExpansion & echo 3L: %%var1%%, %%var2%%, !var1!, !var2!) | more
(cmd /v:on /c echo 4L: %%var1%%, %%var2%%, !var1!, !var2!) | more
cmd /v:on /c echo 5L: %%var1%%, %%var2%%, !var1!, !var2! | more
@endlocal
@echo(
@echo Delayed expansion is now OFF
(cmd /v:on /c echo 6L: %%var1%%, %%var2%%, !var1!, !var2!) | more
cmd /v:on /c echo 7L: %%var1%%, %%var2%%, !var1!, !var2! | more

@setlocal enableDelayedExpansion
@echo(
@echo PIPE RIGHT SIDE - delayed expansion is ON
echo junk | echo 1R: %%var1%%, %%var2%%, !var1!, !var2!
echo junk | (echo 2R: %%var1%%, %%var2%%, !var1!, !var2!)
echo junk | (setlocal enableDelayedExpansion & echo 3R: %%var1%%, %%var2%%, !var1!, !var2!)
echo junk | (cmd /v:on /c echo 4R: %%var1%%, %%var2%%, !var1!, !var2!)
echo junk | cmd /v:on /c echo 5R: %%var1%%, %%var2%%, !var1!, !var2!
@endlocal
@echo(
@echo Delayed expansion is now OFF
echo junk | (cmd /v:on /c echo 6R: %%var1%%, %%var2%%, !var1!, !var2!)
echo junk | cmd /v:on /c echo 7R: %%var1%%, %%var2%%, !var1!, !var2!

Here is the output

NO PIPE - delayed expansion is ON

C:\test>echo 1: value1, , !var1!, !var2!
1: value1, , value1,

C:\test>(echo 2: value1, , !var1!, !var2! )
2: value1, , value1,

PIPE LEFT SIDE - Delayed expansion is ON

C:\test>echo 1L: %var1%, %var2%, !var1!, !var2!   | more
1L: value1, %var2%, value1,


C:\test>(echo 2L: %var1%, %var2%, !var1!, !var2! )  | more
2L: value1, %var2%, !var1!, !var2!


C:\test>(setlocal enableDelayedExpansion   & echo 3L: %var1%, %var2%, !var1!, !var2! )  | more
3L: value1, %var2%, !var1!, !var2!


C:\test>(cmd /v:on /c echo 4L: %var1%, %var2%, !var1!, !var2! )  | more
4L: value1, %var2%, value1, !var2!


C:\test>cmd /v:on /c echo 5L: %var1%, %var2%, !var1!, !var2!   | more
5L: value1, %var2%, value1,


Delayed expansion is now OFF

C:\test>(cmd /v:on /c echo 6L: %var1%, %var2%, !var1!, !var2! )  | more
6L: value1, %var2%, value1, !var2!


C:\test>cmd /v:on /c echo 7L: %var1%, %var2%, !var1!, !var2!   | more
7L: value1, %var2%, value1, !var2!


PIPE RIGHT SIDE - delayed expansion is ON

C:\test>echo junk   | echo 1R: %var1%, %var2%, !var1!, !var2!
1R: value1, %var2%, value1,

C:\test>echo junk   | (echo 2R: %var1%, %var2%, !var1!, !var2! )
2R: value1, %var2%, !var1!, !var2!

C:\test>echo junk   | (setlocal enableDelayedExpansion   & echo 3R: %var1%, %var2%, !var1!, !var2! )
3R: value1, %var2%, !var1!, !var2!

C:\test>echo junk   | (cmd /v:on /c echo 4R: %var1%, %var2%, !var1!, !var2! )
4R: value1, %var2%, value1, !var2!

C:\test>echo junk   | cmd /v:on /c echo 5R: %var1%, %var2%, !var1!, !var2!
5R: value1, %var2%, value1,

Delayed expansion is now OFF

C:\test>echo junk   | (cmd /v:on /c echo 6R: %var1%, %var2%, !var1!, !var2! )
6R: value1, %var2%, value1, !var2!

C:\test>echo junk   | cmd /v:on /c echo 7R: %var1%, %var2%, !var1!, !var2!
7R: value1, %var2%, value1, !var2!

I tested both the left and right side of the pipe to demonstrate that processing is symmetric on both sides.

Tests 1 and 2 demonstrate that parentheses don't have any impact on delayed expansion under normal batch circumstances.

Tests 1L,1R: Delayed expansion works as expected. Var2 is undefined, so %var2% and !var2! output demonstrates that the commands are executed in a command line context, and not a batch context. In other words, command line parsing rules are used instead of batch parsing. (see How does the Windows Command Interpreter (CMD.EXE) parse scripts?) EDIT - !VAR2! is expanded in the parent batch context

Tests 2L,2R: The parentheses disable the delayed expansion! Very bizarre and unexpected in my mind. Edit - jeb considers this an MS bug or design flaw. I agree, there doesn't seem to be any rational reason for the inconsistent behavior

Tests 3L,3R: setlocal EnableDelayedExpansion does not work. But this is expected because we are in a command line context. setlocal only works in a batch context.

Tests 4L,4R: Delayed expansion is initially enabled, but parentheses disable it. CMD /V:ON re-enables delayed expansion and everything works as expected. We still have command line context and output is as expected.

Tests 5L,5R: Almost the same as 4L,4R except delayed expansion is already enabled when CMD /V:on is executed. %var2% gives expected command line context output. But !var2! output is blank which is expected in a batch context. This is another very bizarre and unexpected behavior. Edit - actually this makes sense now that I know !var2! is expanded in the parent batch context

Tests 6L,6R,7L,7R: These are analogous to tests 4L/R,5L/R except now delayed expansion starts out disabled. This time all 4 scenarios give the expected !var2! batch context output.

If someone can provide a logical explanation for results of 2L,2R and 5L,5R then I will select that as the answer to my original question. Otherwise I will probably accept this post as the answer (really more of an observation of what happens than an answer) Edit - jab nailed it!

Addendum: In response to jeb's comment - here is more evidence that piped commands within a batch execute in a command line context, not a batch context.

This batch script:

@echo on
call echo batch context %%%%
call echo cmd line context %%%% | more

gives this output:

C:\test>call echo batch context %%
batch context %

C:\test>call echo cmd line context %%   | more
cmd line context %%

Final Addendum

I've added some additional tests and results that demonstrate all the findings so far. I also demonstrate that FOR variable expansion takes place before the pipe processing. Finally I show some interesting side effects of the pipe processing when a multi-line block is collapsed into a single line.

@echo off
cls
setlocal disableDelayedExpansion
set var1=value1
set "var2="
setlocal enableDelayedExpansion

echo on
@echo(
@echo Delayed expansion is ON
echo 1: %%, %%var1%%, %%var2%%, !var1!, ^^^!var1^^^!, !var2!, ^^^!var2^^^!, %%cmdcmdline%% | more
(echo 2: %%, %%var1%%, %%var2%%, !var1!, ^^^!var1^^^! !var2!, %%cmdcmdline%%) | more
for %%a in (Z) do (echo 3: %%a %%, %%var1%%, %%var2%%, !var1!, ^^^!var1^^^! !var2!, %%cmdcmdline%%) | more
(
  echo 4: part1
  set "var2=var2Value
  set var2
  echo "
  set var2
)
(
  echo 5: part1
  set "var2=var2Value
  set var2
  echo "
  set var2
  echo --- begin cmdcmdline ---
  echo %%cmdcmdline%%
  echo --- end cmdcmdline ---
) | more
(
  echo 6: part1
  rem Only this line remarked
  echo part2
)
(
  echo 7: part1
  rem This kills the entire block because the closing ) is remarked!
  echo part2
) | more

Here is the output

Delayed expansion is ON

C:\test>echo 1: %, %var1%, %var2%, !var1!, ^!var1^!, !var2!, ^!var2^!, %cmdcmdline%   | more
1: %, value1, %var2%, value1, !var1!, , !var2!, C:\Windows\system32\cmd.exe  /S /D /c" echo 1: %, %var1%, %var2%, value1, !var1!, , !var2!, %cmdcmdline% "


C:\test>(echo 2: %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )  | more
2: %, value1, %var2%, !var1!, !var1! !var2!, C:\Windows\system32\cmd.exe  /S /D /c" ( echo 2: %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )"


C:\test>for %a in (Z) do (echo 3: %a %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )  | more

C:\test>(echo 3: Z %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )  | more
3: Z %, value1, %var2%, !var1!, !var1! !var2!, C:\Windows\system32\cmd.exe  /S /D /c" ( echo 3: Z %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )"

C:\test>(
echo 4: part1
 set "var2=var2Value
 set var2
 echo "
 set var2
)
4: part1
var2=var2Value
"
var2=var2Value

C:\test>(
echo 5: part1
 set "var2=var2Value
 set var2
 echo "
 set var2
 echo --- begin cmdcmdline ---
 echo %cmdcmdline%
 echo --- end cmdcmdline ---
)  | more
5: part1
var2=var2Value & set var2 & echo
--- begin cmdcmdline ---
C:\Windows\system32\cmd.exe  /S /D /c" ( echo 5: part1 & set "var2=var2Value
var2=var2Value & set var2 & echo
" & set var2 & echo --- begin cmdcmdline --- & echo %cmdcmdline% & echo --- end cmdcmdline --- )"
--- end cmdcmdline ---


C:\test>(
echo 6: part1
 rem Only this line remarked
 echo part2
)
6: part1
part2

C:\test>(echo %cmdcmdline%   & (
echo 7: part1
 rem This kills the entire block because the closing ) is remarked!
 echo part2
) )  | more

Tests 1: and 2: summarize all the behaviors, and the %%cmdcmdline%% trick really helps to demonstrate what is taking place.

Test 3: demonstrates that FOR variable expansion still works with a piped block.

Tests 4:/5: and 6:/7: show interesting side effects of the way pipes work with multi-line blocks. Beware!

I've got to believe figuring out escape sequences within complex pipe scenarios will be a nightmare.

Funny thing! I don't know the answer, what I know is that the pipeline operation have consistent failures in Windows Batch that should not be present in original MS-DOS Batch (if such features could be executed in old MS-DOS Batch), so I suspect that the error was introduced when the new Windows Batch features were developed.

Here are some examples:

echo Value to be assigned | set /p var=

Previous line does NOT assign the value to the variable, so we must fix it this way:

echo Value to be assigned > temp.txt & set /p var=< temp.txt

Another one:

(
echo Value one
echo Value two
echo Value three
) | call :BatchSubroutine

Doesn't work. Fix it this way:

(
echo Value one
echo Value two
echo Value three
) > temp.txt
call :BatchSubroutine < temp.txt

However, this method DO work in certain cases; with DEBUG.COM for example:

echo set tab=9> def_tab.bat
(
echo e108
echo 9
echo w
echo q
) | debug def_tab.bat
call def_tab
echo ONE%tab%TWO

Previous program show:

ONE     TWO

In which cases works and which not? Only God (and Microsoft) may know, but it seems to be related to new Windows Batch features: SET /P command, delayed expansion, code block in parentheses, etc.

EDIT: Asynchronous Batch files

NOTE: I modified this section to correct an error of mine. See my last comment to jeb for details.

As jeb said, the execution of both sides of a pipeline create two asynchronous processes, that made possible to execute asynchronous threads even if START command is not used.

Mainfile.bat:

@echo off
echo Main start. Enter lines, type end to exit
First | Second
echo Main end

First.bat:

@echo off
echo First start

:loop
    set /P first=
    echo First read: %first%
if /I not "%first%" == "end" goto loop
echo EOF

echo First end

Second.bat:

@echo off
echo Second start

:loop
    set /P second=Enter line: 
    echo Second read: %second%
    echo/
if not "%second%" == "EOF" goto loop

echo Second end

We may use this capability to develop a program equivalent to Expect application (working in a similar way of pexpect Phyton module) that could control any interactive program this way:

Input | anyprogram | Output

Output.bat file will achieve the "Expect" part by analysing the output from the program, and Input.bat will achieve the "Sendline" part by providing the input to the program. The backwards communication from Output to Input modules will be achieved via a file with the desired information and a simple semaphore system controlled via the presence/absence of one or two flag files.