Distributed Reliability Testing
Wednesday, February 25, 2009
We want Google Chrome to be as stable as possible. No matter what site you browse to or what you do, Chrome should never crash. A system we call "distributed reliability testing" is one of the main tools we use to help turn that goal into reality.
One of the advantages to being associated with Google is that we have access to a lot of information about the Web, and a lot of computers to test on. About once an hour, our distributed test infrastructure takes the very latest version of Google Chrome in development and uses it to automatically load a large number of the pages that Google has seen are most popular around the world. When it's done, it produces a report like this on the Buildbot waterfall that all our developers (and anyone else) can see:
Results for top 500 web sites:
success: 499; crashes: 0; crash dumps: 0; timeout: 1
Results for top 500 web sites without sandbox:
success: 463; crashes: 0; crash dumps: 0; timeout: 2
Results for extended list of web sites:
success: 99768; crashes: 3; crash dumps: 3; timeout: 463
Here the final test got through a bit over 100,000 pages before stopping to make way for the next build to be tested. And before each Dev, Beta, or Stable channel release, we run with a much larger number of URLs.
In addition, we "fuzz-test" the user interface, automatically performing arbitrary sequences of actions (opening a new tab, pressing the spacebar, opening various dialogs, etc. — a total of more than 30 possible actions). These are also run in our distributed testing architecture, so we can exercise thousands of combinations for each new version of Google Chrome in progress. The same report that shows the page-load results above collects these UI test results too:
Results for automated UI test:
success: 64643; crashes: 0; crash dumps: 0; timeout: 0
This sort of large-scale testing is great for finding crashes that happen only rarely, or that only affect pages that developers wouldn't have visited as part of their haphazard manual testing. By catching a problem right away even if it's very rare, it's easier for developers to figure out what change caused the error and fix it before it ever gets close to showing up in Google Chrome itself.
19 comments:
dflock said...
That's really impressive - doing regression, crash and real-world testing on ~100k web pages per hour, every hour - in a continuous integration environment. Plus fuzzing, plus the perf. testing you've mentioned in the past. No wonder Chrome works so well! Gotta love that Google infrastructure!
February 25, 2009 4:54 PM
joe said...
thanks that was interesting and impressive. it is always nice to read something of this sort. hope you post more of such.
February 25, 2009 5:02 PM
Morsey said...
Just wanted to let you know about the last time chrome crashed with me... i have a lot of favorites (on the toolbar), and some of them carry over so that i have to click on the >> button to see them. I clicked it, and then I'm not sure what I did... i think i ignored it and added clicked the + to get a new tab or i pressed alt or something... i did have like 7 tabs open or something, and a couple of them had some flash applications running
with all the testing that you do, I am amazed as to how can something as tiny as this leak through!
February 25, 2009 5:35 PM
dmorilha said...
It would be nice notifing the sites if they are in compliance with the standarts as well in order to improve the well formed documents throughout the web.
February 26, 2009 6:06 AM
Tyler Reguly said...
Is it possible to see the list of pages that it's being tested against? that's the data that I'd really be interested in.
February 26, 2009 9:20 AM
Stefano said...
But why, chrome has icudt38.dll instead webkit and safari 4 has icutd40.dll,and this, for a long time?
March 3, 2009 6:36 AM
Vicalla.com said...
Well, it doesn't crash but it often freezes. I normally open only a few tabs and dedicate google chrome only for google websites, like google webmaster tools, google analytics, gmail, adwords, adsense,etc. It freezes quite often and I have to kill it via task manager.
My OS is XP3 with intel 1.6 GHz single core + 1.5 GB ram.
March 4, 2009 7:39 PM
Оригамист said...
Does Google Chrome team gonna to make Mouse Gestures (alike in Avant Browser)?
March 4, 2009 11:32 PM
Rê said...
Dear Developers, The Chrome "Import Bookmark" needs more accuracy. It's a totally caos to update from other browsers. The procedure creates various repeated "Imported Bookmarks" folder distinguished by numbers '(x)'. I think it'll be better to Chrome make atualizations automatically and no need to find 'bookmark.htm' on Application Data folders. Thank you.
March 7, 2009 5:26 AM
Mike said...
The "no crashing" is a lie -- and so is the myth that each tab is its own process. If one page crashes the whole browser crashes. So stop lying and saying that the browser won't crash if one tab has issues. It simply isn't true.
I just lost about 10 pages I had open and needed to work on. I'm no longer using Chrome as it has absolutely no stability whatsoever.
By the way I was accessing the following page:
http://www.xbox.com/NR/rdonlyres/E9BBB98E-9C75-4CEA-946C-F9CA85A25A30/0/vidBionicCommandoHi.asx
March 9, 2009 3:21 PM
Michael said...
I like Google Chrome very much the one thing I miss is being able to use Roboform, will there be any development for that?
March 13, 2009 1:02 PM
Michael said...
I like Google Chrome very much the one thing I miss is being able to use Roboform, will there be any development for that?
March 13, 2009 1:04 PM
Matt said...
The software is slick and works apart from one small issue. It doesn't like torrent sites. In IE you have the ability to OPEN a linked file, rather than just save it. This made file association with external programs much better. You to automatically tunnel your .torrent file into a torrent program and miss several steps. Guess I will need to use IE for that.
March 24, 2009 5:20 PM
whimple said...
Testing against the most popular sites is not necessarily a good proving-ground idea. Sites that crash browsers are not likely to be very popular...
November 3, 2009 4:21 PM
olen said...
Unfortunately, the Chrome English spell checker still sucks. For example, it thinks "jobm" is a word. It misses so many misspelled words it's virtually useless. Please get it together. I don't understand why no one at Google, where I assume they use Chrome all the time, has prioritized this as high.
January 22, 2010 8:22 AM
MonCherie said...
I use Roboform alot. Is there a way to make it work on Chrome? I found where it says it will work with Chromium but not Chrome...I thought they are the same?
Anyway, I am loviong it for Facebook! Please get robo on there.
January 25, 2010 4:38 PM
Bay area shirts said...
Having released Chrome 2.0.167.0 the past week, Google also delivered an insight into how it managed to ensure a high degree of stability for its browser. Hard Drive Recovery
March 17, 2010 11:23 AM
mikeqw said...
The software is slick and works apart from one small issue. It doesn't like torrent sites. In IE you have the ability to OPEN a linked file, rather than just save it. This made file association with external programs much better. compare auto insurance quotes, adipex without prescription, cheap auto insurance quotes You to automatically tunnel your .torrent file into a torrent program and miss several steps. Guess I will need to use IE for that.
June 24, 2010 12:41 PM
altan alır said...
Ah good, I was hoping you guys would follow Mozillas adult oyunlar lead on increasing your max payments.sex filmleri Good on you, this makes browsing the web safer for everyone. oyun oyna
July 21, 2010 6:46 AM
Post a Comment