How to run unit tests (MSTest) in parallel?

I am looking for ways to run test suites in parallel.

I am aware of .testrunconfig setting. This allows you to multiplex on the number of CPUs.

I want to run 1000 tests in parallel. This makes sense because I am testing a web service, so 90% of the time spent in a test is waiting for the service to respond.

Any ideas on how to pull this off ? The tests are written for VS, but I am open to running them outside of VS.

Later edit: the Visual Studio test team have added this in VS 2015 Update 1. See Mark Sowul's answer bellow.

Solution 1:

Most of the answers on this page forget to mention that MSTest parallelizes tests in separate assemblies. You have to split your unittests into multiple .dll's to paralelize it.

But! The recent version - MSTest V2 - now CAN parallelize "in-assembly" (yay!) you just need to install a couple of nuget packages in your test project - TestFramework and TestAdapter - like described here https://blogs.msdn.microsoft.com/devops/2018/01/30/mstest-v2-in-assembly-parallel-test-execution/

And then simply add this to your test project

[assembly: Parallelize(Workers = 4, Scope = ExecutionScope.ClassLevel)]

EDIT: You can also disable parallel execution for a specific test using [DoNotParallelize] on a test method.

Solution 2:

You can get up to 5 by using the method from the Visual Studio Team Test Blog

Keep in mind that there may be concurrency issues using this, as MSTest doesn't completely isolate each test (statics carry over, for example, making things interesting for code meant to run once).

(No idea why the limit is 5, but MSTest will not run them in parallel if parallelTestCount is set to more than 5. As per the comments below, this rule apparently changes with Visual Studio 2013)

Solution 3:

Visual Studio 2015 Update 1 adds this. https://docs.microsoft.com/visualstudio/releasenotes/vs2015-update1-vs#misc

For Update 2, there is a UI toggle button in the toolbar at the top of the Test Explorer pane (between the 'grouping' and 'search' boxes).

For Update 1, Set the following in the .runsettings

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <RunConfiguration>
    <MaxCpuCount>0</MaxCpuCount>
   </RunConfiguration>
</RunSettings>

The value for MaxCpuCount has the following semantics:

• ‘n’ (where 1 <= n <= number of cores) : upto ‘n’ processes will be launched.

• ‘n’ of any other value : The number of processes launched will be as many as the available cores on the machine.

Solution 4:

What I found is that C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe will run parallel tests with a .testsettings file which looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="TestSettings1" id="21859d0f-7bdc-4165-b9ad-05fc803c9ee9" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Description>These are default test settings for a local test run.</Description>
  <Deployment enabled="false" />
  <Execution parallelTestCount="8">
    <TestTypeSpecific>
      <UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
        <AssemblyResolution>
          <TestDirectory useLoadContext="true" />
        </AssemblyResolution>
      </UnitTestRunConfig>
    </TestTypeSpecific>
    <AgentRule name="Execution Agents">
    </AgentRule>
  </Execution>
</TestSettings>

Reference can be found here http://msdn.microsoft.com/en-us/library/vstudio/jj155796.aspx