Running Tests in Parallel

Background

Running unit tests in parallel is a new feature in xUnit.net version 2. There are two essential motivations that drove us to not only enable parallelization, but also for it to be a feature that's enabled by default:

  1. As unit testing has become more prevalent, so too have the number of unit tests. It is not unusual for a project to have thousands—or tens of thousands—of unit tests. Developers want the safety of being able to quickly run all these tests before committing their code.
  2. A typical developer machine in 2006 (when we first started working on xUnit.net) had a single or dual core CPU, and perhaps 2 GB of RAM. Today's modern developer machine is likely to have a CPU with 8 virtual cores and between 8 and 16GB of RAM (or more). These CPUs go to waste when only one of them at a time can be assigned to a given task.

There are really two ways to take advantage of all these extra resources: write tests which themselves use parallelization (so that when the system is only running a single test, it still takes advantage of all the resources); or, let the unit test framework run many tests at the same time, to help take advantage of the available resource.

The typical structure of a unit test is to test a single thing in relative isolation. This doesn't give much opportunity for the test itself to become parallelized, unless the code under test is itself parallelized. Therefore, the best way to ensure that unit tests can run at the full speed of the host computer is to run many of them at the same time.

Runners and Test Frameworks

For the purposes of this section, it's important to separate the two actors that participate in running your unit tests.

The first is the runner, which is the program (or third party plugin to a program) that is responsible for looking for one or more test assemblies, and then activating the test frameworks that it finds therein. It generally contains very little knowledge about how the test frameworks work, and instead relies on xunit.runner.utility to do most of the heavy lifting. Through the runner utility library, it can discover test cases and then ask for them to be run. It does not itself understand how this discovery or execution works, but instead relies on the runner utility library to understand those details.

The second is the test framework, which is the code that has the detailed knowledge of how to discover and run unit tests. These libraries are the ones that the unit tests themselves link against, and so those DLLs live along side the unit test code itself. For xUnit.net v1, that is xunit.dll; for v2, it's xunit.core.dll (and, indirectly, xunit.execution.dll).

There is a third player here that does not have any code, but rather contains the abstractions that allow runners and test frameworks to communicate: xunit.abstractions.dll.

Parallelism in Test Frameworks

When we say "Parallelism in Test Frameworks", what we mean specifically is how a test framework may choose to support running tests within a single assembly in parallel with one another. The next section, "Parallelism in Runners", we mean how a test runner may choose to support running test assemblies in parallel against each other.

As mentioned above, parallelism in the test framework is a feature that's new for version 2. Tests written in xUnit.net version 1 cannot be run in parallel against each other in the same assembly, though multiple test assemblies linked against v1 are still able to participate in the runner parallelism feature described in the next sub-section.

The residual of this section describes features only available for test assemblies linked against xUnit.net v2.

Test Collections

How does xUnit.net decide which tests can run against each other in parallel? It uses a concept called test collections to make that decision.

By default, each test class is a unique test collection. Tests within the same test class will not run in parallel against each other. Let's examine a very simple test assembly, one with a single test class:

public class TestClass1
{
    [Fact]
    public void Test1()
    {
        Thread.Sleep(3000);
    }

    [Fact]
    public void Test2()
    {
        Thread.Sleep(5000);
    }
}

When we run this test assembly, we see that the total time spent running the tests is approximately 8 seconds. These two tests are in the same test class, which means that they are in the same test collection, so they cannot be run in parallel against one another.

If we were to put these two tests into separate test classes, like this:

public class TestClass1
{
    [Fact]
    public void Test1()
    {
        Thread.Sleep(3000);
    }
}

public class TestClass2
{
    [Fact]
    public void Test2()
    {
        Thread.Sleep(5000);
    }
}

Now when we run this test assembly, we see that the total time spent running the tests is approximately 5 seconds. That's because Test1 and Test2 are in different test collections, so they are able to run in parallel against one another.

Custom Test Collections

If we need to indicate that multiple test classes should not be run in parallel against one another, then we place them into the same test collection. This is simply a matter of decorating each test class with an attribute that places them into the same uniquely named test collection:

[Collection("Our Test Collection #1")]
public class TestClass1
{
    [Fact]
    public void Test1()
    {
        Thread.Sleep(3000);
    }
}

[Collection("Our Test Collection #1")]
public class TestClass2
{
    [Fact]
    public void Test2()
    {
        Thread.Sleep(5000);
    }
}

This instructs xUnit.net not run these two classes against each other in parallel. Our total run time now goes back to approximately 8 seconds, which indicates that the tests did indeed run one after another.

For more information on test collections, including the ability to use them to share text context, see Shared Context.

Changing Default Behavior

There are several default pieces of behavior that be configured by the developer as relates to running tests in parallel. These are all changed by applying an assembly level attribute:

You can only have one of these attributes per assembly; if you want to combine multiple behaviors, do it with a single attribute. Also be aware that these values affect only this assembly; if multiple assemblies are running in parallel against one another, they have their own independent values.

Parallelism in Runners

When we say "Parallism in Runners", what we mean specifically is how a runner may choose to run multiple test assemblies in parallel against each other. The decision to do this is independent of whether or not any individual test assembly is running tests within itself in parallel. Runners are also allowed to override some of the behavior within a test framework (like number of threads, whether an assembly should run tests within itself in parallel, etc.); when they do so, it overrides whatever behavior has been otherwise specified (via code or configuration).

There are many runners, and just as many ways to configure parallelism values. This section will cover how to specify parallelism values for the built-in console and MSBuild runners; you should look at the documentation for any third party runners to learn how to configure them.

Console Runner

The console runner in xUnit.net v2 is capable of running unit tests from both xUnit.net v1 and v2. It can run multiple assemblies at the same time, and command line options can be used to configuration the parallelism options used when running the tests.

The following command line options can be used to influence parallelism:

OptionAffect
-parallel option Allows the user to specify which kinds of parallelization should be allowed for the test run. The valid option values are:
none Turns off all parallelization
collections Parallelizes collections, not assemblies
assemblies Parallelizes assemblies, not collections
all Parallelizes both collections and assemblies
The default value is collections.
Applies to: xUnit.net v1, v2
-maxthreads n Overrides the maximum number of threads used per assembly. The default value is determined by the test framework (for xUnit.net v2, it is the number of virtual CPUs in the PC).
Applies to: xUnit.net v2

MSBuild Runner

The MSBuild runner in xUnit.net v2 is capable of running unit tests from both xUnit.net v1 and v2. It can run multiple assemblies at the same time, and build file options can be used to configuration the parallelism options used when running the tests.

The following Xunit task properties can be used to influence parallelism:

PropertyAffect
ParallelizeAssemblies Set to true to run the test assemblies in parallel against one other; set to false to run them sequentially. The default value is false.
Applies to: xUnit.net v1, v2
ParallelizeTestCollections Set to true to run the test collections in parallel against one other; set to false to run them sequentially. The default value is true.
Applies to: xUnit.net v2
MaxParallelThreads Overrides the maximum number of threads used per assembly. The default value is determined by the test framework (for xUnit.net v2, it is the number of virtual CPUs in the PC).
Applies to: xUnit.net v2

Parallelism via Configuration

There are several configuration elements that can influence parallelism. Please see Configuring xUnit.net for more information on how to set up configuration files and change parallelism settings.

Copyright © 2017 .NET Foundation. Contributions welcomed at https://github.com/xunit/xunit.github.io.