Chromium Blog: Distributed Reliability Testing

Chromium Blog

News and developments from the open source browser project

Distributed Reliability Testing

Wednesday, February 25, 2009

We want Google Chrome to be as stable as possible. No matter what site you browse to or what you do, Chrome should never crash. A system we call "distributed reliability testing" is one of the main tools we use to help turn that goal into reality.
One of the advantages to being associated with Google is that we have access to a lot of information about the Web, and a lot of computers to test on. About once an hour, our distributed test infrastructure takes the very latest version of Google Chrome in development and uses it to automatically load a large number of the pages that Google has seen are most popular around the world. When it's done, it produces a report like this on the Buildbot waterfall that all our developers (and anyone else) can see:
Results for top 500 web sites:success: 499; crashes: 0; crash dumps: 0; timeout: 1
Results for top 500 web sites without sandbox:success: 463; crashes: 0; crash dumps: 0; timeout: 2
Results for extended list of web sites:success: 99768; crashes: 3; crash dumps: 3; timeout: 463
Here the final test got through a bit over 100,000 pages before stopping to make way for the next build to be tested. And before each Dev, Beta, or Stable channel release, we run with a much larger number of URLs.
In addition, we "fuzz-test" the user interface, automatically performing arbitrary sequences of actions (opening a new tab, pressing the spacebar, opening various dialogs, etc. — a total of more than 30 possible actions). These are also run in our distributed testing architecture, so we can exercise thousands of combinations for each new version of Google Chrome in progress. The same report that shows the page-load results above collects these UI test results too:
Results for automated UI test:success: 64643; crashes: 0; crash dumps: 0; timeout: 0
This sort of large-scale testing is great for finding crashes that happen only rarely, or that only affect pages that developers wouldn't have visited as part of their haphazard manual testing. By catching a problem right away even if it's very rare, it's easier for developers to figure out what change caused the error and fix it before it ever gets close to showing up in Google Chrome itself.
Posted by Pamela Greene and Patrick Johnson, Software Engineers

Google

Labels

$200K 1
10th birthday 4
abusive ads 1
abusive notifications 2
accessibility 3
ad blockers 1
ad blocking 2
advanced capabilities 1
android 2
anti abuse 1
anti-deception 1
background periodic sync 1
badging 1
benchmarks 1
beta 83
better ads standards 1
billing 1
birthday 4
blink 2
browser 2
browser interoperability 1
bundles 1
capabilities 6
capable web 1
cds 1
cds18 2
cds2018 1
chrome 35
chrome 81 1
chrome 83 2
chrome 84 2
chrome ads 1
chrome apps 5
Chrome dev 1
chrome dev summit 1
chrome dev summit 2018 1
chrome dev summit 2019 1
chrome developer 1
Chrome Developer Center 1
chrome developer summit 1
chrome devtools 1
Chrome extension 1
chrome extensions 3
Chrome Frame 1
Chrome lite 1
Chrome on Android 2
chrome on ios 1
Chrome on Mac 1
Chrome OS 1
chrome privacy 4
chrome releases 1
chrome security 10
chrome web store 32
chromedevtools 1
chromeframe 3
chromeos 4
chromeos.dev 1
chromium 9
cloud print 1
coalition 1
coalition for better ads 1
contact picker 1
content indexing 1
cookies 1
core web vitals 2
csrf 1
css 1
cumulative layout shift 1
custom tabs 1
dart 8
dashboard 1
Data Saver 3
Data saver desktop extension 1
day 2 1
deceptive installation 1
declarative net request api 1
design 2
developer dashboard 1
Developer Program Policy 2
developer website 1
devtools 13
digital event 1
discoverability 1
DNS-over-HTTPS 4
DoH 4
emoji 1
emscriptem 1
enterprise 1
extensions 27
Fast badging 1
faster web 1
features 1
feedback 2
field data 1
first input delay 1
Follow 1
fonts 1
form controls 1
frameworks 1
fugu 2
fund 1
funding 1
gdd 1
google earth 1
google event 1
google io 2019 1
google web developer 1
googlechrome 12
harmful ads 1
html5 11
HTTP/3 1
HTTPS 4
iframes 1
images 1
incognito 1
insecure forms 1
intent to explain 1
ios 1
ios Chrome 1
issue tracker 3
jank 1
javascript 5
lab data 1
labelling 1
largest contentful paint 1
launch 1
lazy-loading 1
lighthouse 2
linux 2
Lite Mode 2
Lite pages 1
loading interventions 1
loading optimizations 1
lock icon 1
long-tail 1
mac 1
manifest v3 2
metrics 2
microsoft edge 1
mixed forms 1
mobile 2
na 1
native client 8
native file system 1
New Features 5
notifications 1
octane 1
open web 4
origin trials 2
pagespeed insights 1
pagespeedinsights 1
passwords 1
payment handler 1
payment request 1
payments 2
performance 20
performance tools 1
permission UI 1
permissions 1
play store 1
portals 3
prefetching 1
privacy 2
privacy sandbox 4
private prefetch proxy 1
profile guided optimization 1
progressive web apps 2
Project Strobe 1
protection 1
pwa 1
QUIC 1
quieter permissions 1
releases 3
removals 1
rlz 1
root program 1
safe browsing 2
Secure DNS 2
security 36
site isolation 1
slow loading 1
sms receiver 1
spam policy 1
spdy 2
spectre 1
speed 4
ssl 2
store listing 1
strobe 2
subscription pages 1
suspicious site reporter extension 1
TCP 1
the fast and the curious 23
TLS 1
tools 1
tracing 1
transparency 1
trusted web activities 1
twa 2
user agent string 1
user data policy 1
v8 6
video 2
wasm 1
web 1
web apps 1
web assembly 2
web developers 1
web intents 1
web packaging 1
web payments 1
web platform 1
web request api 1
web vitals 1
web.dev 1
web.dev live 1
webapi 1
webassembly 1
webaudio 3
webgl 7
webkit 5
WebM 1
webmaster 1
webp 5
webrtc 6
websockets 5
webtiming 1
writable-files 1
yerba beuna center for the arts 1

Feed

Give us feedback in our Product Forums.

Google
Privacy
Terms

Chromium Blog

Distributed Reliability Testing

Labels

Archive

Feed