Testing Chromium: SyzyASAN, a lightweight heap error detector

Monday, May 20, 2013

AddressSanitizer (ASAN) is a tool for finding memory problems and has been used to find thousands of memory errors in Chromium over the last two years. These kinds of errors will typically lead to heap or data corruption and subsequent crashes in random, unrelated code, which make them quite challenging to find and fix without tools like ASAN. However, ASAN is built using LLVM/Clang and is limited to Mac and Linux builds of Chromium. To address the lack of coverage for Windows-only code, we built SyzyASAN.

SyzyASAN is built on top of the Syzygy toolchain and is an instrumentation-based clone of ASAN for detecting heap errors. It consists of three parts:

  • An instrumenter that injects instrumentation into binaries produced by the Microsoft Visual Studio toolchain.
  • A run-time library that replaces malloc, free, et al.
  • An RPC-based logging server that receives information about detected errors. This lets us get information safely out of sandboxed processes, like Chromium’s renderer.
SyzyASAN operates nearly identically to ASAN, finding errors in the same manner and producing similar reports. SyzyASAN finds some of the hardest-to-locate memory bugs like use-after-free, buffer overruns, and underruns. Focusing on very common memory errors allows SyzyASAN to be relatively efficient.

Although Chrome with SyzyASAN is very usable, the penalties in speed - 4.7x on CPU intensive operations - and memory - a 25% increase plus a fixed 256MB increase in each process - are noticeable so we’ll confine these releases to our Canary channel for now. We’ve been releasing SyzyASAN-instrumented builds to the Windows Canary channel one day each week recently. One day with a little slowdown on the Canary channel gives us plenty of great data. In the last three weeks, we’ve found 150 new bugs in Chromium, several of which could lead to security vulnerabilities.

We’ve put together some instructions for instrumenting your local build and debugging issues. Try it out and help us squash more memory bugs. The Syzygy source code and binaries can be downloaded from our code site, and instructions for how to use it are on our wiki. If you have any questions, suggestions or contributions, feel free to contact syzygy-team@chromium.org. If you’re using Syzygy or SyzyASAN with your project we’d love to hear about it!

Post a Comment