Unit testing a multithreaded application?

Does anyone have any advice for a consistent way to unit test a multithreaded application? I have done one application where our mock "worker threads" had a thread.sleep with a time that was specified by a public member variable. We would use this so we could set how long a particular thread would take to complete its work, then we could do our assertions. Any ideas of a better way to do this? Any good mock frameworks for .Net that can handle this?

Solution 1:

Step one is to realise that often much of the code you need to unit test is orthogonal to the threading. This means that you should try and break up the code that does the work from the code that does the threading. Once that's done, you can easily test the code that does the work using normal unit testing practices. But of course, you knew that.

The problem is then one of testing the threading side of the problem, but at least now you have a point where this threading interfaces with the code that does the work and hopefully you have an interface there that you can mock. Now that you have a mock for the code that the threading code calls into, I find the best thing to do is add some events to the mock (this may mean you need to hand roll your mock). The events will then be used to allow the test to synchronise with and block the threading code under test.

So, for example, let's say we have something really simple, a multi-threaded queue that processes work items. You'd mock the work item. The interface might include a 'Process()' method that the thread calls to do the work. You'd put two events in there. One that the mock sets when Process() is called and one that the mock waits on after it has set the first event. Now, in your test you can start up your queue, post a mock work item and then wait on the work item's "I'm being processed" event. If all you're testing is that process gets called, then you can set the other event and let the thread continue. If you're testing something more complex, like how the queue handles multiple dispatch or something, then you might do other things (like post and wait for other work items) before releasing the thread. Since you can wait with a timeout in the test, you can make sure that (say) only two work items get processed in parallel, etc, etc. The key thing is that you make the tests deterministic using events that the threaded code blocks on so that the test can control when they run.

I'm sure your situation is more complex, but this is the basic method that I use to test threaded code and it works pretty well. You can take a surprising amount of control over multi-threaded code if you mock out the right bits and put synchronisation points in.

Here is some more info on this kind of thing, though it's talking about a C++ codebase: http://www.lenholgate.com/blog/2004/05/practical-testing.html

Solution 2:

My advice would be not to rely on unit tests to detect concurrency issues for several reasons:

• Lack of reproducibility: the tests will fail only once in a while, and won't be really helpful to pinpoint the problems.
• Erratic failing build will annoy everybody in the team - because the last commit will always be wrongly suspected for being the cause of the failing build.
• Deadlocks when encountered are likely to freeze the build until the execution timeout is encountered which can significantly slow down the build.
• The build environment is likely to be a single CPU environment (think build being run in a VM) where concurrency issues may never happen - no matter how much sleeping time is set.
• It defeats somehow the idea of having simple, isolated units of validating code.

Solution 3:

If you have to test that a background thread does something, a simple technique I find handy is to to have a WaitUntilTrue method, which looks something like this:

bool WaitUntilTrue(Func<bool> func,
              int timeoutInMillis,
              int timeBetweenChecksMillis)
{
    Stopwatch stopwatch = Stopwatch.StartNew();

    while(stopwatch.ElapsedMilliseconds < timeoutInMillis)
    {
        if (func())
            return true;
        Thread.Sleep(timeBetweenChecksMillis);
    }   
    return false;
}

Used like this:

volatile bool backgroundThreadHasFinished = false;
//run your multithreaded test and make sure the thread sets the above variable.

Assert.IsTrue(WaitUntilTrue(x => backgroundThreadHasFinished, 1000, 10));

This way you don't have to sleep your main testing thread for a long time to give the background thread time to finish. If the background doesn't finish in a reasonable amount of time, the test fails.