Benchmarks are incredibly important for influencing the direction of JavaScript engines. Over the past two years, JavaScript has gotten dramatically faster across all major browsers, particularly in areas that popular benchmarks measure. As it turns out, browser developers are a competitive bunch. However, as JS engines get faster, benchmarks must evolve in order to keep pushing them in the right direction.

We’ve been constantly updating our V8 benchmark suite to force us to get faster in areas that are important to web developers. We’ve fixed bugs and added new tests for regular expressions and memory management. We’re very focused on JavaScript performance, so we scrutinize our benchmark carefully to make sure that it’s as useful a measuring stick as possible.

The two other widely cited JS benchmarks are SunSpider from Apple, and Kraken, a new benchmark from Mozilla.

SunSpider was one of the first suites of tests, first appearing in 2007. It’s done a lot to help improve JS engines, but many of the tests in the suite are less relevant to the web in 2011. Even for the more relevant tests, JavaScript has gotten so fast that many finish in only a few milliseconds. This just isn’t long enough to figure out which engine is faster--a golf cart and a Tesla will finish a 10-foot race in nearly the same time.

To make the benchmark more meaningful, we’ve experimented by making the race longer by running each of the tests in SunSpider 50 times consecutively. While repeating a trivial test many times isn’t a great solution, it does provide an opportunity for some optimization. With this change, the results begin to reflect Chrome’s true performance. It’s more than 30% faster on the SunSpider suite overall and up to 4x faster on some of the individual tests.

Kraken, a more modern benchmark, is in better shape. Unfortunately, the canonical version of the benchmark has not been updated to reflect all the latest changes which address at least one important bug. As a result, the benchmark is less useful and has even (mis)led us to spend time making some irrelevant optimizations in Chrome. To help us focus on the right things, we’re now testing the latest version of Kraken built directly from Mozilla’s source code repository.

We’re posting a modified version of SunSpider and the latest version of Kraken to make it easy to run the benchmarks in your own browser and compare results.

Posted by Mike Belshe, Software Engineer