research-article

Exploring the use of crowdsourcing to support empirical studies in software engineering

Authors:

Kathryn T. Stolee,

Sebastian ElbaumAuthors Info & Claims

ESEM '10: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

Article No.: 35, Pages 1 - 4

https://doi.org/10.1145/1852786.1852832

Published: 16 September 2010 Publication History

Abstract

The power and the generality of the findings obtained through empirical studies are bounded by the number and type of participating subjects. In software engineering, obtaining a large number of adequate subjects to evaluate a technique or tool is often a major challenge. In this work we explore the use of crowdsourcing as a mechanism to address that challenge by assisting in subject recruitment. More specifically, through this work we show how we adapted a study to be performed under an infrastructure that not only makes it possible to reach a large base of users but it also provides capabilities to manage those users as the study is being conducted. We discuss the lessons we learned through this experience, which illustrate the potential and tradeoffs of crowdsourcing software engineering studies.

References

[1]

Julie S. Downs, Mandy B. Holbrook, Steve Sheng, and Lorrie Faith Cranor. Are your participants gaming the system?: screening Mechanical Turk workers. In Proceedings of the 28th international conference on Human factors in computing systems, 2010.

Digital Library

[2]

Brynn M. Evans and Ed H. Chi. Towards a model of understanding social search. In Proceedings of the 2008 ACM conference on Computer supported cooperative work, 2008.

Digital Library

[3]

Martin Fowler and Kent Beck. Refactoring: Improving the Design of Existing Code. Addison-Wesley, 1999.

Digital Library

[4]

Jeffrey Heer and Michael Bostock. Crowdsourcing graphical perception: using Mechanical Turk to assess visualization design. In Proceedings of the 28th international conference on Human factors in computing systems, 2010.

Digital Library

[5]

Jeff Howe. The rise of crowdsourcing. Wired Magazine, 14(06):17--23, 2006.

[6]

M. Cameron Jones and Elizabeth F. Churchill. Conversations in Developer Communities: A Preliminary Analysis of the Yahoo! Pipes Community. In Proceedings of the Fourth International Conference on Communities and Technologies, 2009.

Digital Library

[7]

Aniket Kittur, Ed H. Chi, and Bongwon Suh. Crowdsourcing user studies with Mechanical Turk. In Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, 2008.

Digital Library

[8]

Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. Turkit: tools for iterative tasks on Mechanical Turk. In Proceedings of the ACM SIGKDD Workshop on Human Computation, 2009.

Digital Library

[9]

W. Mason and D. J. Watts. Financial Incentives and the Performance of Crowds. In Proceedings of the ACM SIGKDD Workshop on Human Computation, pages 77--85. ACM, 2009.

Digital Library

[10]

Amazon Mechanical Turk. https://www.mturk.com/mturk/welcome, June 2010.

[11]

Yahoo! Pipes. http://pipes.yahoo.com/, July 2009.

[12]

Joel Ross, Lilly Irani, M. Six Silberman, Andrew Zaldivar, and Bill Tomlinson. Who are the crowdworkers?: shifting demographics in Mechanical Turk. In Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems, 2010.

Digital Library

[13]

Christopher Scaffidi, Mary Shaw, and Brad Myers. Estimating the numbers of end users and end user programmers. In Symposium on Visual Languages and Human Centric Computing, 2005.

Digital Library

[14]

Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2008.

Digital Library

[15]

Kathryn T. Stolee. Analysis and transformation of pipe-like web mashups for end user programmers. Master's Thesis, University of Nebraska-Lincoln, June 2010.

[16]

Survey Vault. http://www.surveyvault.co.uk/, July 2010.

Cited By

Khan HAli FGhadi YNazir SUllah IMohamed H(2023)Human–Computer Interaction and Participation in Software CrowdsourcingElectronics10.3390/electronics1204093412:4(934)Online publication date: 13-Feb-2023
https://doi.org/10.3390/electronics12040934
Feldmeier PStraubinger PFraser GCoppola RArdito LFraser GLeotta M(2023)PlayTest: A Gamified Test Generator for GamesProceedings of the 2nd International Workshop on Gamification in Software Development, Verification, and Validation10.1145/3617553.3617884(47-51)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3617553.3617884
Fronchetti FShepherd DWiese ITreude CGerosa MSteinmacher IChandra SBlincoe KTonella P(2023)Do CONTRIBUTING Files Provide Information about OSS Newcomers’ Onboarding Barriers?Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616288(16-28)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616288
Show More Cited By

Index Terms

Exploring the use of crowdsourcing to support empirical studies in software engineering
1. General and reference
  1. Cross-computing tools and techniques
    1. Metrics

Recommendations

Ethical Issues in Empirical Studies of Software Engineering

The popularity of empirical methods in software engineering research is on the rise. Surveys, experiments, metrics, case studies, and field studies are examples of empirical methods used to investigate both software engineering processes and products. ...
Combining education, industry, and empirical studies in Software Engineering: an experience report
ECSAW '15: Proceedings of the 2015 European Conference on Software Architecture Workshops

Software industry is one of the most pervasive industries today and has a great impact on our day-to-day lives. At the same time, the quality of software systems is directly related to the quality of software engineers -- it is the responsibility of ...
Replication of Empirical Studies in Software Engineering: Preliminary Findings from a Systematic Mapping Study
RESER '11: Proceedings of the 2011 Second International Workshop on Replication in Empirical Software Engineering Research

Our goal in this study is to review the research related to the replication of empirical studies in software engineering in terms of replications of empirical studies and conceptual or theoretical work about replications. In this article we present the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '10: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

September 2010

423 pages

ISBN:9781450300391

DOI:10.1145/1852786

General Chair:
Giancarlo Succi
Free University of Bolzano, Italy
,
Program Chairs:
Maurizio Morisio
Politecnico di Torino, Italy
,
Nachi Nagappan
Microsoft Research

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 September 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ESEM '10

Sponsor:

SIGSOFT

ESEM '10: 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

September 16 - 17, 2010

Bolzano-Bozen, Italy

Acceptance Rates

ESEM '10 Paper Acceptance Rate 30 of 102 submissions, 29%;

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

61
Total Citations
View Citations
924
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)1

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Khan HAli FGhadi YNazir SUllah IMohamed H(2023)Human–Computer Interaction and Participation in Software CrowdsourcingElectronics10.3390/electronics1204093412:4(934)Online publication date: 13-Feb-2023
https://doi.org/10.3390/electronics12040934
Feldmeier PStraubinger PFraser GCoppola RArdito LFraser GLeotta M(2023)PlayTest: A Gamified Test Generator for GamesProceedings of the 2nd International Workshop on Gamification in Software Development, Verification, and Validation10.1145/3617553.3617884(47-51)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3617553.3617884
Fronchetti FShepherd DWiese ITreude CGerosa MSteinmacher IChandra SBlincoe KTonella P(2023)Do CONTRIBUTING Files Provide Information about OSS Newcomers’ Onboarding Barriers?Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616288(16-28)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616288
Davis MAghayi ELatoza TWang XMyers BSunshine J(2023)What’s (Not) Working in Programmer User Studies?ACM Transactions on Software Engineering and Methodology10.1145/358715732:5(1-32)Online publication date: 24-Jul-2023
https://dl.acm.org/doi/10.1145/3587157
Malviya-Thakur AMockus AZaretzki RBichescu BBradley R(2023)How R Developers explain their Package Choice: A Survey2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1109/ESEM56168.2023.10304869(1-12)Online publication date: 26-Oct-2023
https://doi.org/10.1109/ESEM56168.2023.10304869
Lenarduzzi VDieste OFucci DVegas SLanubile F(2021)Towards a Methodology for Participant Selection in Software Engineering ExperimentsProceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3475716.3484273(1-6)Online publication date: 11-Oct-2021
https://dl.acm.org/doi/10.1145/3475716.3484273
Aljohani AJones J(2021)Conducting Malicious Cybersecurity Experiments on Crowdsourcing PlatformsProceedings of the 2021 3rd International Conference on Big Data Engineering10.1145/3468920.3468942(150-161)Online publication date: 29-May-2021
https://dl.acm.org/doi/10.1145/3468920.3468942
Jacques JKristensson P(2021)Studying Programmer Behaviour at Scale: A Case Study using Amazon Mechanical TurkCompanion Proceedings of the 5th International Conference on the Art, Science, and Engineering of Programming10.1145/3464432.3464436(36-48)Online publication date: 22-Mar-2021
https://dl.acm.org/doi/10.1145/3464432.3464436
Silveira FAvancini Rde Souza França DGuerra Eda Silva T(2021)Towards an Extensible Architecture for an Empirical Software Engineering Computational PlatformComputational Science and Its Applications – ICCSA 202110.1007/978-3-030-87013-3_18(231-246)Online publication date: 10-Sep-2021
https://doi.org/10.1007/978-3-030-87013-3_18
Sabou MWinkler DBiffl S(2020)Empirical Software Engineering Experimentation with Human ComputationContemporary Empirical Methods in Software Engineering10.1007/978-3-030-32489-6_7(173-215)Online publication date: 28-Aug-2020
https://doi.org/10.1007/978-3-030-32489-6_7
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents