research-article

Towards a framework to support large scale sampling in software engineering surveys

Authors:

Rafael Maiani de Mello,

Pedro Correa da Silva,

Per Runeson,

Guilherme Horta TravassosAuthors Info & Claims

ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Article No.: 48, Pages 1 - 4

https://doi.org/10.1145/2652524.2652567

Published: 18 September 2014 Publication History

Get Access

Abstract

Context: The low quality and small size of samples in empirical studies in software engineering hamper the interpretation and generalization of their results. Therefore, enlarging sample sizes and improving their quality represent an important research challenge. Goal: We aim to define a conceptual framework, including requirements for establishing adequate sources for sampling subjects in software engineering surveys. Method: We use previous experience on applying systematic sampling strategies combined with contemporary web technologies in previously executed surveys, to organize the conceptual framework. We analyze its application to different sources of sampling. Results: The framework was observed to be feasible after its application to nine different large-scale sources of sampling. Conclusions: The analyzed crowdsourcing tools do not support essential requirements to be considered sources of sampling, while free-lancing tools and professional social network do.

References

[1]

Sjøberg, D. I. et al. 2005. A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering, 31(9): 733--753.

Digital Library

Google Scholar

[2]

Dybå, T., Kampenes, V. B. and Sjøberg, D. I. K. 2007. A systematic review of statistical power in software engineering experiments. Inf. and Soft. Technology 48: 745--755.

Crossref

Google Scholar

[3]

Conradi, R. et al. 2005. Reflections on conducting an international survey of Software Engineering. International Symposium on Empirical Software Engineering, 214--223.

Google Scholar

[4]

Pickard, L. M., Kitchenham, B. A. and Jones, P. W. 1998. Combining empirical results in software Engineering. Information and Software Technology 40: 811--821.

Digital Library

Google Scholar

[5]

de Mello, R. M and Travassos, G. H. 2013. An Ecological Perspective Towards the Evolution of Quantitative Studies in Software Engineering. In: Proc. of 17^th EASE, 216--219.

Digital Library

Google Scholar

[6]

de Mello, R. M. and Travassos, G. H. 2013. Would Sociable Software Engineers Observe Better? In: Proc. of 7th ESEM, 279--282, IEEE.

Google Scholar

[7]

de Mello, R. M., da Silva, P. C. and Travassos, G. H. 2014. Investigating Probabilistic Sampling Approaches for Large-Scale Surveys in Software Engineering. In: 11^th Workshop on Experimental Software Engineering (ESELAW).

Google Scholar

[8]

Wohlin, C. et al. 2012. Experimentation in Software Engineering. Springer.

Digital Library

Google Scholar

[9]

Frakes, W. B., and Fox, C. J. 1995. Sixteen questions about software reuse. 1995. Communications of the ACM 38: 75.

Digital Library

Google Scholar

[10]

Särndal C. A., Swensson, B., and Wretman, J. 1992. Model Assisted Survey Sampling. Springer.

Google Scholar

[11]

Kitchenham, B. A. et al. 2010. Systematic literature reviews in software engineering -- A tertiary study. Information and Software Technology 52: 792--805.

Digital Library

Google Scholar

[12]

Stolee, K. T. and Elbaum S. 2010. Exploring the Use of Crowdsourcing to Support Empirical Studies in Software Engineering. In: Proc. of 4th ESEM.

Digital Library

Google Scholar

[13]

Stolee, K. T. and Elbaum S. 2013. On the Use of Input/Output Queries for Code Search. In: Proc. 7th ESEM.

Google Scholar

Cited By

View all

Malviya-Thakur AMockus AZaretzki RBichescu BBradley R(2023)How R Developers explain their Package Choice: A Survey2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1109/ESEM56168.2023.10304869(1-12)Online publication date: 26-Oct-2023
https://doi.org/10.1109/ESEM56168.2023.10304869
Lenarduzzi VDieste OFucci DVegas SLanubile F(2021)Towards a Methodology for Participant Selection in Software Engineering ExperimentsProceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3475716.3484273(1-6)Online publication date: 11-Oct-2021
https://dl.acm.org/doi/10.1145/3475716.3484273
Ore JDetweiler CElbaum S(2021)An Empirical Study on Type AnnotationsACM Transactions on Software Engineering and Methodology10.1145/343977530:2(1-29)Online publication date: 10-Feb-2021
https://dl.acm.org/doi/10.1145/3439775
Show More Cited By

Index Terms

Towards a framework to support large scale sampling in software engineering surveys
1. Theory of computation
  1. Semantics and reasoning
    1. Program reasoning
      1. Program verification

Recommendations

Surveys in Software Engineering: Identifying Representative Samples
ESEM '16: Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Context: The representativeness of samples in Software Engineering primary studies is still a great challenge, mainly when identifying available sources for establishing adequate sampling frames, characterizing subjects and stimulating their ...
Sampling improvement in software engineering surveys
ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Context: Small and non-probabilistic samples represent relevant issues when discussing the external validity of empirical studies in Software Engineering. Goal: To investigate alternatives to improve the quality of samples (size, heterogeneity and level ...
Sampling in software engineering research: a critical review and guidelines
Abstract
Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

September 2014

461 pages

ISBN:9781450327749

DOI:10.1145/2652524

General Chair:
Maurizio Morisio
Politecnico di Torino, Italy
,
Program Chairs:
Tore Dybå
Sintef, Norway
,
Marco Torchiano
Politecnico di Torino, Italy

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 September 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conselho Nacional de Desenvolvimento Científico e Tecnológico

Conference

ESEM '14

Sponsor:

SIGSOFT

ESEM '14: 2014 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

September 18 - 19, 2014

Torino, Italy

Acceptance Rates

ESEM '14 Paper Acceptance Rate 23 of 123 submissions, 19%;

Overall Acceptance Rate 130 of 594 submissions, 22%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
262
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)0

Reflects downloads up to 04 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Malviya-Thakur AMockus AZaretzki RBichescu BBradley R(2023)How R Developers explain their Package Choice: A Survey2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1109/ESEM56168.2023.10304869(1-12)Online publication date: 26-Oct-2023
https://doi.org/10.1109/ESEM56168.2023.10304869
Lenarduzzi VDieste OFucci DVegas SLanubile F(2021)Towards a Methodology for Participant Selection in Software Engineering ExperimentsProceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3475716.3484273(1-6)Online publication date: 11-Oct-2021
https://dl.acm.org/doi/10.1145/3475716.3484273
Ore JDetweiler CElbaum S(2021)An Empirical Study on Type AnnotationsACM Transactions on Software Engineering and Methodology10.1145/343977530:2(1-29)Online publication date: 10-Feb-2021
https://dl.acm.org/doi/10.1145/3439775
Garousi VFelderer MKuhrmann MHerkiloğlu KEldh S(2020)Exploring the industry's challenges in software testingJournal of Software: Evolution and Process10.1002/smr.225132:8Online publication date: 3-Aug-2020
https://dl.acm.org/doi/10.1002/smr.2251
Garousi VPfahl DFernandes JFelderer MMäntylä MShepherd DArcuri ACoşkunçay ATekinerdogan B(2019)Characterizing industry-academia collaborations in software engineeringEmpirical Software Engineering10.1007/s10664-019-09711-y24:4(2540-2602)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.1007/s10664-019-09711-y
Ore JElbaum SDetweiler CKarkazis LHuchard MKästner CFraser G(2018)Assessing the type annotation burdenProceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering10.1145/3238147.3238173(190-201)Online publication date: 3-Sep-2018
https://dl.acm.org/doi/10.1145/3238147.3238173
de Mello RTravassos GGenero MJedlitschka AJørgensen MScanniello GSampath SCaivano DPort D(2016)Surveys in Software EngineeringProceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/2961111.2962632(1-6)Online publication date: 8-Sep-2016
https://dl.acm.org/doi/10.1145/2961111.2962632
Molléri JPetersen KMendes EGenero MJedlitschka AJørgensen MScanniello GSampath SCaivano DPort D(2016)Survey Guidelines in Software EngineeringProceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/2961111.2962619(1-6)Online publication date: 8-Sep-2016
https://dl.acm.org/doi/10.1145/2961111.2962619
França BTravassos G(2016)Experimentation with dynamic simulation models in software engineeringEmpirical Software Engineering10.1007/s10664-015-9386-421:3(1302-1345)Online publication date: 1-Jun-2016
https://dl.acm.org/doi/10.1007/s10664-015-9386-4
de Mello Rda Silva PTravassos G(2015)Investigating probabilistic sampling approaches for large-scale surveys in software engineeringJournal of Software Engineering Research and Development10.1186/s40411-015-0023-03:1Online publication date: 10-Jun-2015
https://doi.org/10.1186/s40411-015-0023-0
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Surveys in Software Engineering: Identifying Representative Samples

Sampling improvement in software engineering surveys

Sampling in software engineering research: a critical review and guidelines

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations