Monday, April 21, 2014

Testing Chromium: ThreadSanitizer v2, a next-gen data race detector

In 2009, we developed ThreadSanitizer (aka TSan), a runtime data race detector based on binary translation. The tool helped find thousands of threading errors in various projects, including almost 180 bugs in Chromium. In 2010, we started experimenting with compiler-based instrumentation instead of binary translation, and once the approach had proven itself, our team redesigned ThreadSanitizer from scratch, focusing on compile-time instrumentation for greater speed and accuracy.

The new tool, ThreadSanitizer v2, is now part of both LLVM and GCC. Not only is it able to detect data races in C++ and Go code, but it is also able to report synchronization issues like deadlocks, unjoined threads, destroying locked mutexes, use of async-signal unsafe code in signal handlers, and others.

ThreadSanitizer v2 recognizes atomic operations, so it produces far more accurate reports and can also find bugs in lock-free algorithms, which are hard to detect otherwise. For example, it has helped us find incorrect synchronization in LazyInstance, incorrect lock-free queue implementation, and issues with concurrent sweeping in V8.

The new version is up to 20 times faster than the old ThreadSanitizer, which makes it possible to use it for heavyweight browser tests. Thanks to this, we’ve found several bugs in code that wasn’t covered by smaller unit tests, like races on the WebKit strings, garbage collection in Blink, a use-after-free in Blink, a handful of reports in WebRTC and many more. This has also improved bug reproducibility: the more times the test is repeated, the higher the chance to reproduce a flaky bug.

Over the last half-year almost 100 bugs were detected by the new tool, and we’re actively working on more. Our future plans include extensive use of TSan on ClusterFuzz and adding regular testing for various Chromium subprojects to catch new regressions quickly.

The ThreadSanitizer page contains all of the information necessary to start using the tool in your project. The tool is easy to use and can be integrated with any buildsystem: just add a single compile-time flag and run the program to see the error reports. For Chromium developers there’s a special page with instructions on

Posted by Alexander Potapenko, Software Engineer and Synchronization Lookout