Freezing Threads in the Debugger

Once you know which threads are involved in a race condition, you can begin looking for the variable that isn't protected by a lock, and once you know that, the rest is easy. But if the bug only appears sporadically, how can you force the system to make the bug appear? How can you make the threads run in a predictable fashion so you can verify theories? Suppose you want to see what happens if one thread yields control in a certain spot—how can you test this? There's only one way to gain complete control over which threads run and which threads sleep: Use the thread debugging window.

Manually controlling the execution of each thread is tedious, but it gets the job done. Set a breakpoint on the code that starts each new thread and run the program. As each new thread is started, freeze it with the thread debugging window. Now you control how the threads run. Select the thread you care about, and use the debugger to run that thread for as long as you like. Once it reaches a spot that seems tricky (such as just before it's about to acquire some lock or modify some variable), freeze that thread and start another. You control exactly when each threads switches, so you can theoretically reproduce any possible race condition or deadlock that your code contains.

One time I had to debug a program in which a certain variable became corrupted—we suspected a race condition where some code somewhere modified the variable without first acquiring the lock. Now how to prove that? At first I thought I could simply search for all the places where that variable was used to see if they were lock protected. But that didn't work because many of those places had deliberately not been lock protected under the theory that the calling function would already hold the lock. So instead, I froze all threads but one, ran that one thread to the point where it acquired the lock, and then froze the thread. Next I set a conditional breakpoint to alert me when the variable changed, unfroze all the other threads, and waited. Soon, my conditional breakpoint fired—but since I knew that should have been impossible when my frozen thread held the lock protecting the variable, I had my answer.

Using the thread debugging window to control the ordering of threads is great when you have a theory you want to test or when you want to run a random experiment in hopes of getting more data. But even though it lets you run the threads in whatever order you want, it doesn't advise you on what that order should be. You're on your own for that part. Still, if nothing else, this technique allows you to debug your program as if it were single threaded, and step across each line of code one at a time. Never underestimate the power of that.

Tracking Threading Bugs with Logging

Product logging seems low tech compared to VS .NET's advanced debugger, but it often succeeds where the debugger can't. Logging is a great way to track down race conditions and understand the order threads are running in. Frequently, you'll notice variables with inconsistent values and you'll want to know how the threads switched to create that situation. The debugger won't be able to tell you since it slows the code down too much to reproduce the issue. But with a bunch of debug-only log statements, you can see when two threads are overlapping when they shouldn't.

Thread #2 about to read x
Thread #1 about to modify x
Thread #3 about to modify y
Thread #1 finished with x
Thread #3 finished with y
Thread #2 finished with x

Reading logs like that may take some concentration, but it does reveal that thread #2 is reading x at the same time thread #1 is writing it. That shouldn't be possible—thread #2 ought to acquire a lock before reading x so that thread #1 can't get in until #2 has finished, and this is probably the key to your race condition. Anytime you see overlap of two threads with the same variable, then you should investigate because it usually means you need additional synchronization.

Of course, that type of logging produces hundreds of lines that get in the way when you're hunting down other bugs. Remember log levels from Chapter 5? Situations like this might be a good place to employ those so you can easily turn the multithreaded logging on and off. Alternatively, some developers prefer to add those log statements only in the debug builds as they're needed and remove them once the bug is fixed. I'd recommend the former approach, but the choice is up to you.

Always Acquire Locks in the Same Order

All the deadlock examples we've seen in this chapter have involved one thread acquiring two locks in the order A-B, while another thread acquires the locks in the order B-A. That lets both threads get hung waiting on the other.

class LockOrderTest {
   static LockOrderTest A = new LockOrderTest();
   static LockOrderTest B = new LockOrderTest();
   //First thread runs this function
   static void FirstThread() {
      //Acquire locks in the order A-B
      lock(A) {
         lock(B) {
             ...
         }
      }
}
   //Second thread runs this function
   static void SecondThread() {
     //Acquire locks in the order B-A
     lock(B) {
        lock(A) {
            ...
        }
     }
   }
}

But suppose both threads acquired the locks in the order A-B? Suppose we left the FirstThread function as-is and rewrote the SecondThread function to be

static void SecondThread() {
   lock(A) {
      lock(B) {
          ...
      }
   }
}

Then one thread or the other would get the A lock, and the second thread would wait since the A lock was unavailable. But now nothing is blocking the first thread from also acquiring the B lock, doing its business, and then releasing both locks. As long as you're careful to always acquire all locks in the same order, then it's impossible to ever have a deadlock, no matter how many threads and locks are involved.

The main downside to this method is that it's sometimes hard to control. In sample code, all the locks are acquired in a single method, and in those cases, acquiring locks in the same order is easy. But suppose the process of acquiring locks is spread out over several methods? Function A acquires lock 1, function B acquires lock 2, and so on. You may have some difficulty keeping track of which functions call which other functions and which ones acquire which locks. As a result, you may plan to acquire locks in the order 1-2-3-4, but you might forget that a function that acquires lock 3 makes a call to a function that acquires lock 2. Always acquiring locks in the same order can solve the deadlock problem, but only if you're careful.

Use Only One Lock

This idea is so obvious that many developers never even seriously consider it, but one extremely manageable way to eliminate deadlocks is to use only a single lock in your program. If there's only one lock, then any thread that acquires the lock is guaranteed to eventually finish its business and release the lock, which allows some other thread to grab it. There is no possible way a deadlock can ever occur in this situation.

This idea is far more practical than you might at first think. You never need more than one lock to guarantee proper thread synchronization; in fact, a single lock makes the development substantially easier because there are fewer things for you to track. One of the most successful multithreaded programs I've ever been involved with made use of exactly one lock, and this decision was a big part of that success. The strategy of having only a single lock is so simple that many developers may reject it, but it deserves serious, serious consideration.

The one drawback to using a single lock is that it limits the amount of multi-threading that can take place. To get the absolute maximum amount of multithreading, you would have a different lock for every single variable that the threads access. Under that approach, a dozen threads could simultaneously modify a dozen unrelated variables protected by a dozen different locks, and there would be no problem as long as no two threads tried to modify the same variable. By contrast, under the single lock approach, only one of those dozen threads could modify any variable at any given time. In the absolute worst case, a single-lock application would essentially behave almost as if it were single threaded because no two threads would ever be able to run simultaneously.

Will that worst case come about very often? It depends on your application, but usually not. If you need absolute maximum performance, then a single lock may not be the way to go. But for "good-enough" performance, single locks will very often suffice; and they'll definitely shorten your debugging cycle.

Use Multiple-Reader, Single-Writer Locks

So far in this chapter, we've exclusively discussed only the lock and SyncLock methods of thread synchronization because they're the easiest. But these methods aren't as powerful as some of the other synchronization techniques. In particular, the lock and SyncLock statements allow only a single thread to access code at a time, and that may be more restrictive than you need. Many times, it's actually perfectly fine to let multiple threads access the same variable at once. But in order to do it safely, you need to use a different synchronization technique.

Think back to our example of a race condition. Two threads simultaneously ran the following code:

using System.Threading;
class MyTestClass {
   public static int x = 0;
   public static void DoUselessStuff() {
      while (true) {
         int m = x;
         x++;
         System.Diagnostics.Debug.Assert ( m + 1 == x );
      }
   }
   public static void Main() {
      Thread t1 = new Thread(new ThreadStart(DoUselessStuff));
      Thread t2 = new Thread(new ThreadStart(DoUselessStuff));
      t1.Start();
      t2.Start();
   }
}

This created a race condition because it let one thread write to the x variable while the other thread read from it, and that allowed one thread to work with a stale copy of the variable. But what would have happened if both threads had merely read the x variable, without writing to it? Would there have been a problem? No. As long as both threads are read-only, then the x variable never gets out of date, and there's no race condition. For truly read-only variables, you don't need synchronization at all! In fact, synchronization actually hurts in that situation, partly because it reduces the number of threads that can run simultaneously and also because the unnecessary locks provide one more opportunity for a deadlock.

Well, knowing that synchronization isn't necessary for read-only variables doesn't help us much because most real-world variables aren't read-only. However, most of them are read far more often than they're written, and we can use a different type of synchronization to take advantage of that. Recall that a standard lock allows access to only one thread at a time, but .NET also provides System.Threading.ReaderWriterLock, which allows access to an arbitrary number of threads simultaneously, but only if they are read-only. A thread that wants to write must wait for all current read-only threads to release their lock (no new reading threads will be allowed in while a thread is waiting for a write lock), and then that thread will have the variable to itself.

using System.Threading;
class ReaderWriterTest {
    private ReaderWriterLock rwLock = new ReaderWriterLock();
    private int x = 0;

   //Any number of threads could be in this function at once
   void Func1() {
      rwLock.AcquireReaderLock(0);
      ...//Do read-only stuff with x
      rwLock.ReleaseReaderLock();
   }
   //only one thread can get in this function at a time
   void Func2() {
     rwLock.AcquireWriterLock(0);
     x++;
     rwLock.ReleaseWriterLock();
   }
}

The downside of ReaderWriterLock is that it's not automatic. You have to manually release the lock when you're done. On the other hand, it does provide two big advantages. The first is performance related—multiple threads don't do any good if most of the threads are waiting at any given time, and the ReaderWriterLock allows multiple read-only threads to proceed at once. Under the right circumstances, that can yield much better performance than a standard lock.

The other advantage of ReaderWriterLock is that it reduces (but doesn't completely eliminate) the likelihood of a deadlock. Any thread that acquires the reader lock could potentially cause a deadlock with a thread waiting for a writer lock— that's no better and no worse than a standard lock. But at least two reader threads can no longer deadlock each other, and that should reduce the number of opportunities for disaster. You'll still accidentally code a deadlock now and then, but the law of averages says it'll happen less often.