Can Powershell Run Commands in Parallel?

I have a powershell script to do some batch processing on a bunch of images and I'd like to do some parallel processing. Powershell seems to have some background processing options such as start-job, wait-job, etc, but the only good resource I found for doing parallel work was writing the text of a script out and running those (PowerShell Multithreading)

Ideally, I'd like something akin to parallel foreach in .net 4.

Something pretty seemless like:

foreach-parallel -threads 4 ($file in (Get-ChildItem $dir))
{
   .. Do Work
}

Maybe I'd be better off just dropping down to c#...

Solution 1:

You can execute parallel jobs in Powershell 2 using Background Jobs. Check out Start-Job and the other job cmdlets.

# Loop through the server list
Get-Content "ServerList.txt" | %{

  # Define what each job does
  $ScriptBlock = {
    param($pipelinePassIn) 
    Test-Path "\\$pipelinePassIn\c`$\Something"
    Start-Sleep 60
  }

  # Execute the jobs in parallel
  Start-Job $ScriptBlock -ArgumentList $_
}

Get-Job

# Wait for it all to complete
While (Get-Job -State "Running")
{
  Start-Sleep 10
}

# Getting the information back from the jobs
Get-Job | Receive-Job

Solution 2:

The answer from Steve Townsend is correct in theory but not in practice as @likwid pointed out. My revised code takes into account the job-context barrier--nothing crosses that barrier by default! The automatic $_ variable can thus be used in the loop but cannot be used directly within the script block because it is inside a separate context created by the job.

To pass variables from the parent context to the child context, use the -ArgumentList parameter on Start-Job to send it and use param inside the script block to receive it.

cls
# Send in two root directory names, one that exists and one that does not.
# Should then get a "True" and a "False" result out the end.
"temp", "foo" | %{

  $ScriptBlock = {
    # accept the loop variable across the job-context barrier
    param($name) 
    # Show the loop variable has made it through!
    Write-Host "[processing '$name' inside the job]"
    # Execute a command
    Test-Path "\$name"
    # Just wait for a bit...
    Start-Sleep 5
  }

  # Show the loop variable here is correct
  Write-Host "processing $_..."

  # pass the loop variable across the job-context barrier
  Start-Job $ScriptBlock -ArgumentList $_
}

# Wait for all to complete
While (Get-Job -State "Running") { Start-Sleep 2 }

# Display output from all jobs
Get-Job | Receive-Job

# Cleanup
Remove-Job *

(I generally like to provide a reference to the PowerShell documentation as supporting evidence but, alas, my search has been fruitless. If you happen to know where context separation is documented, post a comment here to let me know!)

Solution 3:

There's so many answers to this these days:

1. jobs (or threadjobs in PS 6/7 or the module for PS 5)
2. start-process
3. workflows (PS 5 only)
4. powershell api with another runspace
5. invoke-command with multiple computers, which can all be localhost (have to be admin)
6. multiple session (runspace) tabs in the ISE, or remote powershell ISE tabs
7. Powershell 7 has a foreach-object -parallel as an alternative for #4

Using start-threadjob in powershell 5.1. I wish this worked like I expect, but it doesn't:

echo yahoo.com facebook.com | 
  start-threadjob { test-netconnection $input } | receive-job -wait -auto

WARNING: Name resolution of yahoo.com microsoft.com facebook.com failed

It works this way. Not quite as nice and foreach-object -parallel in powershell 7 but it'll do.

echo yahoo.com facebook.com | 
  % { $_ | start-threadjob { test-netconnection $input } } | 
  receive-job -wait -auto | ft -a

ComputerName RemotePort RemoteAddress PingSucceeded PingReplyDetails (RTT) TcpTestS
                                                                           ucceeded
------------ ---------- ------------- ------------- ---------------------- --------
facebook.com 0          31.13.71.36   True          17 ms                  False
yahoo.com    0          98.137.11.163 True          97 ms                  False

Here's workflows with literally a foreach -parallel:

workflow work {
  foreach -parallel ($i in 1..3) { 
    sleep 5 
    "$i done" 
  }
}

work

3 done
1 done
2 done

Or a workflow with a parallel block:

function sleepfor($time) { sleep $time; "sleepfor $time done"}

workflow work {
  parallel {
    sleepfor 3
    sleepfor 2
    sleepfor 1
  }
  'hi'
}
    
work 

sleepfor 1 done
sleepfor 2 done
sleepfor 3 done
hi

Here's an api with runspaces example:

$a =  [PowerShell]::Create().AddScript{sleep 5;'a done'}
$b =  [PowerShell]::Create().AddScript{sleep 5;'b done'}
$c =  [PowerShell]::Create().AddScript{sleep 5;'c done'}
$r1,$r2,$r3 = ($a,$b,$c).begininvoke() # run in background
$a.EndInvoke($r1); $b.EndInvoke($r2); $c.EndInvoke($r3) # wait
($a,$b,$c).streams.error # check for errors
($a,$b,$c).dispose() # clean

a done
b done
c done