V8 Benchmark Suite Updated

Thursday, October 07, 2010

Labels: ,

The V8 benchmark suite contains a number of pure JavaScript benchmarks that capture the areas in which a JavaScript engine has to perform well to support the well-structured, maintainable, and high-performance web applications of tomorrow. These benchmarks have been useful for us when optimizing the V8 JavaScript engine and we have found that making them run faster leads to better performance for many of the web applications we enjoy using every day.

Today we have released version 6 of the V8 benchmark suite. The main changes are in the RegExp and Splay components of the benchmark suite. For reference, we describe each of the existing benchmarks in the suite below, along with any changes made in version 6.

RegExp: Regular expression benchmark generated by extracting regular expression operations from 50 of the most popular web pages. The regular expressions are exercised a number of times to reflect their popularity on those top 50 web pages. Changed in version 6: each regular expression is now exercised on a number of different input strings instead of just one.
Splay: Data manipulation benchmark that modifies a large splay tree to exercise the automatic memory management subsystem. The benchmark builds a large splay tree in a setup phase and then measures how fast nodes can be added and removed. Changed in version 6: no longer converts the same numeric key to string repeatedly and updates the splay tree in a way that increases the pressure on the memory management subsystem.

Richards: Operating system kernel simulation benchmark originally written in BCPL by Martin Richards. The Richards benchmark effectively measures how fast the JavaScript engine is at accessing object properties, calling functions, and dealing with polymorphism. It is a standard benchmark that has been successfully used to measure the performance of many modern programming language implementations.

DeltaBlue: One-way constraint solver, originally written in Smalltalk by John Maloney and Mario Wolczko. The DeltaBlue benchmark is written in an object-oriented style with a multi-level class hierarchy. As such it measures how fast the JavaScript engine is at running well-structured applications with many objects and small functions. Changed in version 6: fixed a couple of typos that do not have any impact on the behavior of the benchmark.

Crypto: Encryption and decryption benchmark based on code by Tom Wu. The benchmark encrypts an input string, decrypts the result and verifies that encryption followed by decryption yields the original input. The encryption/decryption algorithm is RSA and the benchmark measures the performance of arithmetic operations on integers and array access.

RayTrace: Ray tracer benchmark based on code by Adam Burmister. The benchmark measures floating-point computations where the object structure is constructed using the Prototype JavaScript library. Changed in version 6: removed dead code that has no impact on the behavior of the benchmark.

EarleyBoyer: Classic Scheme benchmarks, translated to JavaScript by Florian Loitsch's Scheme2Js compiler. The benchmarks exercise important areas of the JavaScript engine such as object allocation, data structure manipulation, and garbage collection. The translated nature of the benchmarks make them appear foreign, but the runtime characteristics of the benchmarks are highly representative of many real world web applications.

Curious to know how your browser performs? Give it a spin on the new version of the V8 benchmark suite.

12 comments:

rumours said...

chrome 7.0.544.0 dev
Score: 2470

opera 10.62
Score: 1775

firefox 4.0b6
Score: 522

firefox 3.6.10
Score: 264

ie 8 - fail

rumours said...

restart

chrome 7.0.544.0 dev
Score: 2516

opera 10.62
Score: 1946

ie 9 beta
Score: 697

T.J. Crowder said...

Very cool!

FYI, getting some markedly strange results on a Windows XP VM with Chrome. Whereas my Windows 7 VM and its host (Ubuntu) both give reasonable-looking numbers (4,820 in 21 seconds and 4,014 in 20 seconds, respectively), a Windows XP VM of the same spec on the same host comes back with a whopping 30,227 — but takes 115 seconds to do so. That just doesn't make sense to me, thought I should mention it. I don't have a non-VM Windows XP box to try it on, though.

FWIW, here are the full results (raw CSV) for my Windows XP VM, Windows 7 VM, and their Ubuntu 10.04 LTS host (an Intel i3-520 ~3GHz 4GB RAM).

Browsers tested:
* Chrome
* Opera
* Firefox
* IE7 (on Windows XP only)
* IE8 (on Windows 7 only)

On:
* Ubuntu 10.04 LTS
* Windows XP (VM)
* Windows 7 (VM)

Summary:
* V8 rocks
* Opera's engine is surprisingly agile
* SpiderMonkey could use some love
* JScript is a dog

No great surprises there, other than the odd XP result for Chrome.

Keep up the great work!

T.J. Crowder said...

Sorry, the "raw CSV" link should have been http://pastie.org/1207625.txt.

Erik Corry said...

T.J.: I think there's something wrong with the clock in your virtual machine. If you bring up a clock application, does it tick at the right speed?

T.J. Crowder said...

@Erik: Yes, it does (curiously, I just recently had reason to be paying attention to that, for something else completely). And while the speeds for Firefox and Opera are different for that VM than the other, they're still in the same ballpark. Chrome's numbers go into a neighbouring franchise. :-)

But yeah, barring someone with a "real" Windows XP system chiming in with the same sort of oddity, I think we have to assume it's an artifact of the VM, whether the VM behaves normally otherwise or not (and it does).

Speaking of counting seconds: It'd be handy to see how long the test took, in total, in the results display.

leeight said...

Score: 5259
Richards: 5415
DeltaBlue: 4846
Crypto: 5808
RayTrace: 6492
EarleyBoyer: 15669
RegExp: 1947
Splay: 3683

News Dabbler said...

My browser results were
Score: 4905
Richards: 5246
DeltaBlue: 5845
Crypto: 5797
RayTrace: 5550
EarleyBoyer: 15710
RegExp: 1853
Splay: 2377

Is this ok? My shockwave has been crashing everyday with chrome dev 555.0

Alex said...

Impressive. Chrome has really changed the game when it comes to javascript performance:
chrome 7 2513

The latest firefox is catching up though firefox 4 pre-b8:
2201

Aldo said...

IE8: 32.3

I don't feel like running V8 Benchmark in IE6 xD!

praxis22 said...

Chromium 7.0.517.44 (0) Dev
Score: 4710
Richards: 3823
DeltaBlue: 5316
Crypto: 5420
RayTrace: 4726
EarleyBoyer: 15042
RegExp: 1863
Splay: 3527

asi said...

I wish Google should also start "Pilot Program" in India as well so that we can also try out the "Cr-48 Chrome notebook".