Chromium Blog: Putting It to the Test

Chromium Blog

News and developments from the open source browser project

Historically, testing hasn't gotten much respect in the world of software development. As the old saying goes, "It compiles! Ship it!" Only a joke — but like most jokes, it hides a grain of truth.

Not so for the Chromium project. Our philosophy is to test everything we possibly can, in as many ways as we can think of.
Test drive: why test?
It's easy to find arguments against testing. Writing tests takes time that developers could be using to write features, and keeping the test hardware and software infrastructure running smoothly isn't trivial. (I'm one of the people largely responsible for the latter for Chromium, along with Nicolas Sylvain, so I know how time-consuming it can be.) But in the long run, it's a big win, for at least two reasons.
A well-established set of tests that developers are expected to run before sending changes in makes it a lot easier to avoid causing problems, which lets other developers stay productive rather than chasing down regressions. And testing submitted changes promptly keeps the code building cleanly and minimizes trouble in the longer term.
But even more importantly, an extensive set of automated tests gives us more confidence that Chromium is reliable, stable, and correct. We're not afraid to rewrite major portions of the code, because verifying correctness afterward is easier. And we have the flexibility to iterate faster and produce releases more often, because we don't need a 6-month QA cycle before each one.
The test of time: performance testing
We run a lot of different tests. Tests of security. Tests of UI functionality. Tests of startup time, page-load speed, DOM manipulation, memory usage. Tests for memory errors using Rational Purify. WebKit's suite of layout tests. Hundreds of unit tests to make sure that individual methods are still doing what they should. At last count, we run more than 9100 individual tests, typically 30-40 times every weekday.^[1] You can find the full list in the developer documentation, but I'll talk more about one broad category here: performance testing.
With every change made in the tree, we keep track of Chromium's page-load time, memory usage, startup time, the time to open a new tab or switch to one, and more. All these data points are available in graphs like this one:

Here the top, gold trace shows the startup time on XP for the tip-of-tree build; the green, bottom trace shows the startup time for a reference build so we can discount variation in the test conditions; and the blue, middle trace shows the startup time along a different code path that includes loading gears.dll. The light blue horizontal line is a reference marker. As you can see, whatever changed between the previous build and r3693, it increased the startup time (gold trace) by more than 8%. The developer responsible was able to see that and fix the problem a few builds later.
This graph also shows the usefulness of running a reference build. The spike in startup time that lasted only a single build also shows up in the reference-build time (the green trace). We can assume that it was something temporarily affecting the build machine, rather than a code change. (The problem must have cleared up by the time the Gears startup test ran.)
With so many performance graphs, it can be hard to watch them all, so there's also a summary page.
One final note about Chromium's performance graphs: they're written in HTML and JavaScript, and we're looking for someone to make them easier to use. If you're interested, grab the code and start hacking!
Test bed: the Chromium buildbot
Nearly all of this testing is controlled by Chromium's buildbot, which automates the build/test cycle. Every time a change is submitted, the buildbot master builds the tree, runs the tests on all the different platforms, and displays the results. For a complete guide to the buildbot and its "waterfall" result page, see the Tour of the Chromium Buildbot in the developer docs.
Pro-test
Of course, once you have lots of tests running, the second important aspect of good tree hygiene is to keep them all passing. But that's a subject for another post.
[1] It's hard to put a single number on it, because certain tests only apply to some parts of the code. But however you count it, it's a lot of tests.
Posted by Pamela Greene, Software Engineer

Google

Labels

$200K 1
10th birthday 4
abusive ads 1
abusive notifications 2
accessibility 3
ad blockers 1
ad blocking 2
advanced capabilities 1
android 2
anti abuse 1
anti-deception 1
background periodic sync 1
badging 1
benchmarks 1
beta 83
better ads standards 1
billing 1
birthday 4
blink 2
browser 2
browser interoperability 1
bundles 1
capabilities 6
capable web 1
cds 1
cds18 2
cds2018 1
chrome 35
chrome 81 1
chrome 83 2
chrome 84 2
chrome ads 1
chrome apps 5
Chrome dev 1
chrome dev summit 1
chrome dev summit 2018 1
chrome dev summit 2019 1
chrome developer 1
Chrome Developer Center 1
chrome developer summit 1
chrome devtools 1
Chrome extension 1
chrome extensions 3
Chrome Frame 1
Chrome lite 1
Chrome on Android 2
chrome on ios 1
Chrome on Mac 1
Chrome OS 1
chrome privacy 4
chrome releases 1
chrome security 10
chrome web store 32
chromedevtools 1
chromeframe 3
chromeos 4
chromeos.dev 1
chromium 9
cloud print 1
coalition 1
coalition for better ads 1
contact picker 1
content indexing 1
cookies 1
core web vitals 2
csrf 1
css 1
cumulative layout shift 1
custom tabs 1
dart 8
dashboard 1
Data Saver 3
Data saver desktop extension 1
day 2 1
deceptive installation 1
declarative net request api 1
design 2
developer dashboard 1
Developer Program Policy 2
developer website 1
devtools 13
digital event 1
discoverability 1
DNS-over-HTTPS 4
DoH 4
emoji 1
emscriptem 1
enterprise 1
extensions 27
Fast badging 1
faster web 1
features 1
feedback 2
field data 1
first input delay 1
Follow 1
fonts 1
form controls 1
frameworks 1
fugu 2
fund 1
funding 1
gdd 1
google earth 1
google event 1
google io 2019 1
google web developer 1
googlechrome 12
harmful ads 1
html5 11
HTTP/3 1
HTTPS 4
iframes 1
images 1
incognito 1
insecure forms 1
intent to explain 1
ios 1
ios Chrome 1
issue tracker 3
jank 1
javascript 5
lab data 1
labelling 1
largest contentful paint 1
launch 1
lazy-loading 1
lighthouse 2
linux 2
Lite Mode 2
Lite pages 1
loading interventions 1
loading optimizations 1
lock icon 1
long-tail 1
mac 1
manifest v3 2
metrics 2
microsoft edge 1
mixed forms 1
mobile 2
na 1
native client 8
native file system 1
New Features 5
notifications 1
octane 1
open web 4
origin trials 2
pagespeed insights 1
pagespeedinsights 1
passwords 1
payment handler 1
payment request 1
payments 2
performance 20
performance tools 1
permission UI 1
permissions 1
play store 1
portals 3
prefetching 1
privacy 2
privacy sandbox 4
private prefetch proxy 1
profile guided optimization 1
progressive web apps 2
Project Strobe 1
protection 1
pwa 1
QUIC 1
quieter permissions 1
releases 3
removals 1
rlz 1
root program 1
safe browsing 2
Secure DNS 2
security 36
site isolation 1
slow loading 1
sms receiver 1
spam policy 1
spdy 2
spectre 1
speed 4
ssl 2
store listing 1
strobe 2
subscription pages 1
suspicious site reporter extension 1
TCP 1
the fast and the curious 23
TLS 1
tools 1
tracing 1
transparency 1
trusted web activities 1
twa 2
user agent string 1
user data policy 1
v8 6
video 2
wasm 1
web 1
web apps 1
web assembly 2
web developers 1
web intents 1
web packaging 1
web payments 1
web platform 1
web request api 1
web vitals 1
web.dev 1
web.dev live 1
webapi 1
webassembly 1
webaudio 3
webgl 7
webkit 5
WebM 1
webmaster 1
webp 5
webrtc 6
websockets 5
webtiming 1
writable-files 1
yerba beuna center for the arts 1

Feed

Give us feedback in our Product Forums.

Google
Privacy
Terms

Chromium Blog

Putting It to the Test

Labels

Archive

Feed