Staged execution of unit tests on Team Build

Using Team Foundation Sever 2015 or newer or are you on Azure DevOps? Find updated guidance here!

When running Unit Tests in your build system, you may want to first run the most important suite, the one that should never fail, the ones that are currently being changed, finally the regression suite and integration tests that may be slower to execute.

Team Build supports this feature, though many probably never knew that this was in the system since the first release of the Workflow based Team Build 2010.

In your build definition you can easily add multiple test runs, and for each test run define which tests should run. Not a fixed list of tests (a vsmdi test list), but a dynamic list based on attributes of the test.

The following steps will take you through the basic gist to get this working on your Team Build instance.

1. Categorize your tests

Step one is to group tests together that need go run together. This can be done in many different ways. Grouping can be done at the assembly or namespace level, yet the most flexible system I tend to rely on is grouping by Test Category.

Go through your test base and apply categories to your automated tests. Use the [TestCategory] attribute for MsTest and the [Category] attribute for NUnit. XUnit is currently unsupported.

// MsTest
[TestCategory("Critical")]
[TestMethod]
public void MyCriticalTest {}
// NUnit
[Category("Critical")]
[Test]
public void MyCriticalTest {}

You can even combine different methods such as Category, Priority and/or naming:

// MsTest
[TestCategory("Critical")]
[TestPriority(1)]
[TestMethod]
public void MyCriticalTest {}

Instead of using categories or attributes, you can also use a naming pattern in the test name, namespace or assembly name, more on this in part 2.

2. Create multiple test runs in Team Build

Once you've categorized your tests create the test runs in Team Build. Each stage or set of tests will map to its own test run. In this example we're going to distinguish critical tests from non-critical tests using a simple attribute.

In your Build definition open the Test Runs screen using this "hidden" glyph. First select the line for the button to appear, then click it:

And create two test runs using the "add" button:

We're going to configure the first run to execute the critical tests (and fail the build when any of these tests fail). Only when these tests succeed will we run the non-critical tests.

Edit the first Test run:

Set the following fields:

• Name: give the run a name, this is reflected in the Code Coverage report and shows up in the build details.
• Fail build on test failure: critical tests in this example should always pass, so check the box.
• Test assembly specification: In this case we're leaving these standard, but when you want to distinguish critical tests based on the assembly name, this is where you'd select a pattern to make that distinction.
• Test Case Filter: In this example we're going to use the TestCategory attribute to distinguish the Critical tests. For now, you'll need to accept the filter we're adding here TestCategory=Critical. I'll explain what options you can enter here further down.

Edit the second run and set:

• Name: give the run a name, this is reflected in the Code Coverage report and shows up in the build details.
• Fail build on test failure: for non-critical tests we want the build to succeed (this will result in a Partial Success)
• Test assembly specification: In this case we're leaving these standard, but when you want to distinguish critical tests based on the assembly name, this is where you'd select a pattern to make that distinction.
• Test Case Filter: In this example we're going to use the TestCategory attribute to distinguish the non-critical tests. For now, you'll need to accept the filter we're adding here TestCategory!=Critical. I'll explain what options you can enter here further down.

Now when you run your build definition it will first run all tests with the "Critical" category and only when this succeeds will it run all tests without this category.

Alternative filters

Test Categories are a very common way to filter tests, but you may have a large set of tests that already follow a different naming pattern. A common distinction is to put all Integration tests in their own assembly or in a different namespace.

This can be accomplished quite easily as well. To understand what filtering options you have available, here is an overview of what you can do.

Assembly name

The Tests Assembly file Specification follows the standard MsBuild Item Specs:

• **/ recursive folders
• * wildcard search
• ? Single character placeholder

When your Integration tests are in their own projects you can use the project and assembly name to select these tests:

• \*IntegrationTests.dll for integration tests
• \*UnitTests.dll for unit tests

Namespace or class

The Test Case Filter option can filter on a number of items, Full name and Classname (not all projects support the classname filter) can be used to filter. You can either use an exact match (=), or a contains filter (~), eg:

• FullyQualifiedName~Integration
• ClassName~Important

The operators and properties are explained here quite well.

Category or Priority

As mentioned above you can use Test Categories, or you can filter on Priority.

• Priority=1

You can also combine multiple items using & (and) and | (or):

• Priority=1&TestCategory=Critical
• Priority=2&TestCategory=Critical
• Priority=3&TestCategory=Critical

This would queue 3 runs, each would run less important tests that are still critical, before running the rest of the tests.

Source: StackOverflow