Any satisfactory approaches to unit testing thread safety in Java?

I am looking at improving a package that I believe not to be threadsafe when its input is shared between multiple worker threads. According to TDD principles, I should write some tests that fail in the first instance, and these would certainly be useful in assessing the problem.

I realise that this is not a simple thing to acheive, and that naively, multi-threaded tests will be nondeterministic as the operating system will determine scheduling and the exact order that various operations are interleaved. I have looked at and used MultithreadedTC in the past, and this was useful. However, in that case I knew in advance exactly where the existing implementation fell down, and thus was able to cook up a nice set of tests that covered it.

However, if you're not at the point where you know exactly what the problem is, is there a good way of going about writing a test that stands a good chance of throwing up any potential problems? Are there any libraries that others have found helpful? Would I be right in thinking that from a purist point of view, a multi-threaded test case should just be the same calls and assertions as the usual single-threaded test, only run with multiple worker threads as appropriate?

Any offers on tools/best practices/philosophy in general would be welcome.

Java Concurrency in Practice has some great information about how to write tests for concurrency issues. However they are not true unit tests. It is nearly impossible to write a true unit test for a concurrency issue.

Basically it boils down to this. Create a bunch of test threads and start them. Each thread should

• wait for a count down latch
• repeatedly call some method that modifies the mutable state in question
• count down on a second latch and exit

The junit thread creates all the threads and starts them, then counts down on the first latch once to let them all go, then waits for the second latch, then makes some assertions about the mutable state.

Even more so than other types of bugs, it's easier to write a failing unit test for a concurrency bug after you have found the bug.

Forget getting good results by testing for concurrency problems. Attempt to reduce synchronization and make the problem smaller. Then use as high as possible library support to do synchronization. And only if you really have then try to handle the concurrency yourself. When you know each worker does its work correct and all your thinking tell you that you have the concurrency problem licked then generate some interesting load. Unittest frameworks and their extensions can do that job, but know that you are not testing any units anymore. (Remember, you already had that part covered)

If your concurrency model gets complicated check out tools suited for that like SPIN.

There are two basic problems you have to solve. The first is this: how do you generate a thread clash? Second, how do you validate your test?

The first one is straightforward. Use a big hammer. Write some code that's capable of detecting the case where one thread steps on another, and run that code 50 times or so on 50 consecutive threads. You can do this with a countdown latch:

public void testForThreadClash() {
  final CountDownLatch latch = new CountDownLatch(1);
  for (int i=0; i<50; ++i) {
    Runnable runner = new Runnable() {
      public void run() {
        try {
          latch.await();
          testMethod();
        } catch (InterruptedException ie) { }
      }
    }
    new Thread(runner, "TestThread"+i).start();
  }
  // all threads are waiting on the latch.
  latch.countDown(); // release the latch
  // all threads are now running concurrently.
}

Your testMethod() has to generate a situation where, without the synchronized block, some thread will step on another thread's data. It should be able to detect this case and throw an exception. (The code above doesn't do this. The exception will most likely be generated on one of the test threads, and you need to put in a mechanism that will detect it on the main thread, but that's a separate issue. I left it out for simplicity, but you can do this with a second latch, which the test can prematurely clear on an exception.)

The second question is trickier, but there's a simple solution. The question is this: How do you know that your hammer will generate a clash? Of course, your code has synchronized blocks to prevent a clash, so you can't answer the question. But you can. Just remove the synchronized keywords. Since it's the synchronized keyword that makes your class threadsafe, you can just remove them and re-run the test. If the test is valid, it will now fail.

When I first wrote the code above, it turned out to be invalid. I never saw a clash. So it's not (yet) a valid test. But now we know how to get a clear answer to the second question. We're getting the wrong answer, but we can now tinker with the test to generate the failure we're looking for. Here's what I did: I just ran the test 100 times in a row.

for (int j=0; j<100; ++j) {
  testForThreadClash();
}

Now my test failed reliably on about the 20th iteration or so. This confirms my test is valid. I can now restore the synchronized keyword and rerun the test, confident that it will tell me if my class is threadsafe.