Things that make your unit tests unreliable

by Jesse Houwing

Over the past few months I've come across a lot of Unit Tests that triggered all kinds of crazy corner cases inside mstest. The most interesting observation is that the behavior inside Visual Studio is often different (or doesn't occur) than in Team Build when this happens. This post will cover some of the issues we've ran into so far.

This most crazy cases all seem to occur when types are loaded using Reflection (either directly by you, or indirectly by the framework) or when you rely on generated proxy classes (WCF's ChannelFactory for example). As long as the code is loaded directly by the Test Runner no issues occur.

In this post:

Incorrectly handled Exceptions from Async methods of a Proxy class can kill Mstest

If you have tests that test Async Code, it might be better to test the contents of the Async method separately and then just test that the Async method is being scheduled. By not testing the actual Async mechanism, this issue won't occur.

I haven't been able to reproduce this issue outside of our solution, I can explain a bit about how it came to be:
The test concentrated around calling an async WCF web service contract from an asynchronously invoked delegate. If the web service throws an error, our little corner case would surface.

I discussed this at length with Peter Provost at the ALM Summit and it is a well known issue with many Test Runners. Mstest will try to catch exceptions thrown from a different thread, it relies on Proxy classes to do so. When you load your classes through Reflection or your classes use their own Proxy Generation, then mstest will not be able to intercept the exception. Normally you can see that an exception has occurred on a different thread. The Test Results window will show a yellow exclamation mark (as seen below)


And when you click the Test test run error link, it will show you the actual error that occurred:

In our case we didn't get a Test Run Error, but instead it brought down mstest and also kept VsPerfMon running, locking the code coverage file. This in turn causes the Test Runner inside Visual Studio to stall (until you kill VsPerfMon.exe and QtAgent.exe).

On the build server the same thing happens, but the build itself won't hang, it is just marked Partially Successful. It's the next build you run in the same workspace, which will start to fail due to the fact that the Code Coverage files are still locked. Killing VsPerfMon and QtAgent.exe on the build agent temporarily fixes the issue.

We resolved the issue by relying completely on WCF to handle the Async call and we removed the Asynchronously invoked delegate from the equation. This actually solved the issue completely.

Throwing exceptions from types that are loaded through Reflection can break code coverage

When building a custom Configuration Section, you often inherit from ConfigurationElement and ConfigurationElementCollection. When you create an instance of these classes, or add an element to them they will actually try to load the app.config from the Test Project. And it will thus run the logic inside it.

If that logic throws an exception, a SecurityException in our case,  it will disconnect VsPerfMon from the running test. Your test will succeed, no test run errors will occur, you will just see partial code coverage results. In our case the first test triggered this case and the code coverage file only contained the line with throw new SecurityException("...");.

We solved this by removing the tags from the app.config, creating a custom ConfigurationManager to run the tests against. This ConfigurationManager can be configured at the start of the test and where needed we use Moles to substitute the original ConfigurationManager with our implementation.

Using Accessors to call into private code is not supported by Team Build

Visual Studio has a very simple framework which helps you to test Private methods without having to change them to public. The documentation already states that this functionality is not guaranteed to carry over into future versions of Visual Studio.


It turns out that generating Accessors is already unsupported on Team Build x64, so while your tests will work just fine inside Visual Studio, they won't in Team Build. Since we're already using Moles for a number of other tests, we've decided to ditch Accessors in favor of Moles (you can also use TypeMock Isolator if you prefer). Rumor has it that Microsoft will to integrate Moles directly into the next release of Visual Studio.

Another workaround is to use the PrivateObject class directly, that is what the Accessors feature uses under the hood, and that works just fine inside Team Build.

For Internal methods we're using InternalsVisibleTo to allow the tests to access Internal/Friend methods and properties. This post from Marko Apfel clearly explains how to get the required information. Using InternalsVisibleTo is a lot faster than using Reflection or one of the above mentioned mocking frameworks.
An even better solution is to refactor the code to allow us to test it properly. Using protected and internal instead of private where applicable.

3.5 Test projects and 4.0 Test projects don't mix well

When you install Visual Studio 2010 SP1, you are allowed to create both 3.5 and 4.0 based test projects. This feature was specifically added for the SharePoint folk and since we have a pretty substantial SharePoint component in most of our projects, we're using them quite extensively. You do need to tweak your .testsettings file to get the tests to work correctly with SharePoint.

The biggest problem is that you can't combine them in one test run. So if you're used to running all your tests inside Visual Studio by pressing CTRL-R, T to run all your tests, you're in for a surprise if your solution contains both test project types. Same applies to Team Build if you're relying on a FileSpec to select all your test assemblies in one go.

We've found a few solutions to this problem

  1. Don't mix 3.5 and 4.0 test projects in one solution. This is the simplest one, everything will work as expected and you might only have to tweak your testsettings file to run the tests in x64 mode if you're testing SharePoint.
  2. Use a Test List  to separate 3.5 and 4.0 tests.
  • Inside Visual Studio: Right-click test list you've created and only run those tests. You will need do this again for the other category.
  • Inside Team Build: don't rely on FileSpecs, but use a Test List to run one category. You can then add a second Automated test (see below) to run the other category in the same build.
  1. Use a naming convention to separate 3.5 tests from 4.0 tests. Just tag "v35" and "v40" to your test project names.
  • Inside Visual Studio: You can use the Project filter in the Test View to select just the 3.5 or just the 4.0 tests:

  • Inside Team Build: You can use the FileSpec to filter on *Test*v35*.dll to run just the 3.5 tests. Then create a second Automated Test and filter on *Test*v40*.dll on that

The option to add an additional Automated Test to Team Build is a well-hidden gem. I've asked around quite a bit before being pointed into this direction.

To add more Test Assemblies, highlight the "Automated Tests" line in the Build Definition Editor, just click the little [...] button to bring up a dialog which allows you to add additional test settings. You can specify a different FileSpec for each of them, configure different Test Lists etc.

We've opted for the solution that uses the naming conventions, it makes the build definitions easy to maintain and we don't have to keep updating the Test List files all the time. We use the [TestCategory("...")] extensively to put the tests in different executable groups, which we then use to either fail the build or just log a warning.

People keep forgetting to update the .testsettings and Test List files.

MsTest relies a lot on lists. You need to create and maintain .testsettings file, you need to specify which assemblies to instrument, you need to create lists of tests and all of these lists must be maintained. People tend to forget to maintain these lists. I'm not pointing fingers, it's just what we're observing.

So, as I mentioned in the previous section, we like to do things by convention. Learning people a convention is just so much easier than learning people to maintain well hidden lists of things. We're using the [TestCategory("...")] attribute extensively as described in this post from Ed Blankenship.

We're using the project name to find out where our tests live. And we're using a custom .target file to allow Instrumentation of our assemblies using a simple MsBuild argument specified on the Build Process's MsBuild Arguments option.

Leave a comment.