research-article

Overlapping experiment infrastructure: more, better, faster experimentation

Authors:

Diane Tang,

Ashish Agarwal,

Deirdre O'Brien,

Mike MeyerAuthors Info & Claims

KDD '10: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining

Pages 17 - 26

https://doi.org/10.1145/1835804.1835810

Published: 25 July 2010 Publication History

Get Access

Abstract

At Google, experimentation is practically a mantra; we evaluate almost every change that potentially affects what our users experience. Such changes include not only obvious user-visible changes such as modifications to a user interface, but also more subtle changes such as different machine learning algorithms that might affect ranking or content selection. Our insatiable appetite for experimentation has led us to tackle the problems of how to run more experiments, how to run experiments that produce better decisions, and how to run them faster. In this paper, we describe Google's overlapping experiment infrastructure that is a key component to solving these problems. In addition, because an experiment infrastructure alone is insufficient, we also discuss the associated tools and educational processes required to use it effectively. We conclude by describing trends that show the success of this overall experimental environment. While the paper specifically describes the experiment system and experimental processes we have in place at Google, we believe they can be generalized and applied by any entity interested in using experimentation to improve search engines and other web applications.

Supplementary Material

JPG File (kdd2010_tang_oei_01.jpg)

Download
13.19 KB

MOV File (kdd2010_tang_oei_01.mov)

Download
126.84 MB

References

[1]

D. Agarwal, A. Broder, D. Chakrabarti, D. Diklic,V. Josifovski, and M. Sayyadian. Estimating rates of rareevents at multiple resolutions. In Proceedings of the ACMConference on Knowledge Discovery and Data Mining(KDD), 2007.

Digital Library

Google Scholar

[2]

W. G. Cochran. Sampling Techniques. Wiley, 1977.

Google Scholar

[3]

D. Cox and N. Reid. The theory of the design of experiments, 2000.

Google Scholar

[4]

T. Crook, B. Frasca, R. Kohavi, and R. Longbotham. Seven pitfalls to avoid when running controlled experiments on the web. Microsoft white paper, March 2008. http://exp-platform.com/Documents/ExPpitfalls.pdf.

Google Scholar

[5]

Google. <a href="http://www.google.com/analytics/siteopt">Google website optimizer.</a>

Google Scholar

[6]

T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD), 2002.

Digital Library

Google Scholar

[7]

R. Kohavi, R. Longbotham, D. Sommerfield, and R. M. Henne. Controlled experiments on the web: Survey and practical guide. Data Mining and Knowledge Discovery, 18, no. 1:140--181, July 2008.

Digital Library

Google Scholar

[8]

M. Krieger. Wrap up & experimentation: Cs147l lecture, 12 2009. http://hci.stanford.edu/courses/cs147/lab/slides/08-experimentation.pdf.

Google Scholar

[9]

Microsoft. <a href='http://exp-platform.com/default.aspx'>Microsoft's experimentation platform. </a>

Google Scholar

[10]

M. Richardson, E. Dominowska, and R. Ragno. Predicting clicks: Estimating the click-through rate for new ads. In Proceedings of the 16th International World Wide Web Conference, 2007.

Digital Library

Google Scholar

[11]

L. Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer Texts, 2004.

Digital Library

Google Scholar

Cited By

View all

Zhao J(2024)Experimental Design through an Optimization LensSSRN Electronic Journal10.2139/ssrn.4780792Online publication date: 2024
https://doi.org/10.2139/ssrn.4780792
Xiong TWang Y(2024)Large-Scale Metric Computation in Online Controlled Experiment PlatformProceedings of the VLDB Endowment10.14778/3685800.368582317:12(4014-4024)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.14778/3685800.3685823
Cockburn AHills DChen ZGutwin C(2024)User Interface Evaluation Through Implicit-Association TestsProceedings of the ACM on Human-Computer Interaction10.1145/36646368:EICS(1-23)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3664636
Show More Cited By

Index Terms

Overlapping experiment infrastructure: more, better, faster experimentation
1. Mathematics of computing
  1. Probability and statistics
    1. Statistical paradigms
      1. Exploratory data analysis

Recommendations

Online controlled experiments at large scale
KDD '13: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining

Web-facing companies, including Amazon, eBay, Etsy, Facebook, Google, Groupon, Intuit, LinkedIn, Microsoft, Netflix, Shop Direct, StumbleUpon, Yahoo, and Zynga use online controlled experiments to guide product development and accelerate innovation. At ...
Trustworthy online controlled experiments: five puzzling outcomes explained
KDD '12: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining

Online controlled experiments are often utilized to make data-driven decisions at Amazon, Microsoft, eBay, Facebook, Google, Yahoo, Zynga, and at many other companies. While the theory of a controlled experiment is simple, and dates back to Sir Ronald ...
Controlled experiments on the web: survey and practical guide

The web provides an unprecedented opportunity to evaluate ideas quickly using controlled experiments, also called randomized experiments, A/B tests (and their generalizations), split tests, Control/Treatment tests, MultiVariable Tests (MVT) and parallel ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

KDD '10: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining

July 2010

1240 pages

ISBN:9781450300551

DOI:10.1145/1835804

General Chairs:
Bharat Rao
Siemens
,
Balaji Krishnapuram
Siemens
,
Program Chairs:
Andrew Tomkins
Google Inc.
,
Qiang Yang
Hong Kong University of Science and Technology

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD '10

Sponsor:

KDD '10: The 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

July 25 - 28, 2010

DC, Washington, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

211
Total Citations
View Citations
2,351
Total Downloads

Downloads (Last 12 months)153
Downloads (Last 6 weeks)25

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Zhao J(2024)Experimental Design through an Optimization LensSSRN Electronic Journal10.2139/ssrn.4780792Online publication date: 2024
https://doi.org/10.2139/ssrn.4780792
Xiong TWang Y(2024)Large-Scale Metric Computation in Online Controlled Experiment PlatformProceedings of the VLDB Endowment10.14778/3685800.368582317:12(4014-4024)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.14778/3685800.3685823
Cockburn AHills DChen ZGutwin C(2024)User Interface Evaluation Through Implicit-Association TestsProceedings of the ACM on Human-Computer Interaction10.1145/36646368:EICS(1-23)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3664636
Wu Jd'Amorim M(2024)AutoOffAB: Toward Automated Offline A/B Testing for Data-Driven Requirement EngineeringCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663780(472-476)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663780
Quin FWeyns DBaresi LMa XPasquale L(2024)Automating Pipelines of A/B Tests with Population Split Using Self-Adaptation and Machine LearningProceedings of the 19th International Symposium on Software Engineering for Adaptive and Self-Managing Systems10.1145/3643915.3644087(84-97)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643915.3644087
Su WDuan WBaeza-Yates RBonchi F(2024)Improving Ego-Cluster for Network Effect MeasurementProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671557(5713-5722)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671557
Mahajan P(2024)Cost-Effective A/B Testing: Leveraging Go and Python for Efficient Experimentation in Hermes Testing Platform2024 10th International Conference on Communication and Signal Processing (ICCSP)10.1109/ICCSP60870.2024.10543437(1048-1050)Online publication date: 12-Apr-2024
https://doi.org/10.1109/ICCSP60870.2024.10543437
Quin FWeyns DGalster MSilva C(2024)A/B testingJournal of Systems and Software10.1016/j.jss.2024.112011211:COnline publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112011
Halder SLim KChan JZhang X(2024)A survey on personalized itinerary recommendation: From optimisation to deep learningApplied Soft Computing10.1016/j.asoc.2023.111200152(111200)Online publication date: Feb-2024
https://doi.org/10.1016/j.asoc.2023.111200
Chen LPei YWan MFei ZLiang TMa G(2024)Smart Issue Detection for Large-Scale Online Service Systems Using Multi-Channel DataFundamental Approaches to Software Engineering10.1007/978-3-031-57259-3_8(165-187)Online publication date: 6-Apr-2024
https://doi.org/10.1007/978-3-031-57259-3_8
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Online controlled experiments at large scale

Trustworthy online controlled experiments: five puzzling outcomes explained

Controlled experiments on the web: survey and practical guide