research-article

Guidelines for snowballing in systematic literature studies and a replication in software engineering

Author:

Claes WohlinAuthors Info & Claims

EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

Article No.: 38, Pages 1 - 10

https://doi.org/10.1145/2601248.2601268

Published: 13 May 2014 Publication History

Abstract

Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.

Objective: This paper presents guidelines for conducting literature reviews using a snowballing approach, and they are illustrated and evaluated by replicating a published systematic literature review.

Method: The guidelines are based on the experience from conducting several systematic literature reviews and experimenting with different approaches.

Results: The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.

Conclusions: It is concluded that using snowballing, as a first search strategy, may very well be a good alternative to the use of database searches.

References

[1]

Kitchenham, B. A., Dybå, T. and Jørgensen, M. 2004. Evidence-based software engineering. In Proceedings of 27th IEEE International Software Engineering Conference, 273--281, IEEE Computer Society, 2004.

Digital Library

[2]

Webster, J. and Watson, R. T. 2002. Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly 26, 2, xiii--xxiii.

Digital Library

[3]

Pickard, L., Kitchenham B. A. and Jones P. 1998. Combining empirical results in software engineering. Information & Software Technology 40, 14, 811--821.

Digital Library

[4]

Miller, J. 1999. Can results from software engineering experiments be safely combined?. In Proceedings IEEE 6th International Symposium on Software Metrics, 152--158.

Digital Library

[5]

Hayes, W. 1999. Research synthesis in software engineering: A case for meta-analysis. In Proceedings 6th IEEE International Software Metrics Symposium, 143--151.

Digital Library

[6]

Basili V. R., Shull F. and Lanubile F. 1999. Building knowledge through families of experiments. IEEE Transactions on Software Engineering 25, 4, 456--473.

Digital Library

[7]

Kitchenham B. A. and Charters S. 2007. Guidelines for performing systematic literature reviews in software engineering. Version 2.3, EBSE Technical Report, EBSE-2007-01, Keele University.

[8]

Petersen K., Feldt R., Mujtaba S. and Mattsson M. 2008. Systematic mapping studies in software engineering. In Proceedings 12^th International Conference on Evaluation and Assessment in Software Engineering.

Digital Library

[9]

Kitchenham B. A., Budgen, D. and Brereton, O. P. 2011. Using mapping studies as the basis for further research -- a participant-observer case study. Information and Software Technology 53, 6, 638--651.

Digital Library

[10]

Wohlin C. and Prikladnicki R. 2013. Systematic literature reviews in software engineering. Information and Software Technology 55, 6, 919--920.

Digital Library

[11]

MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676--687.

Digital Library

[12]

Wohlin, C., Runeson, P., da Mota Silveira Neto, P. A., Engström, E., do Carmo Machado, I. and de Almeida, E. S. 2013. On the reliability of mapping studies in software engineering. Journal of Systems and Software 86, 10, 2594--2610.

[13]

Kitchenham, B. A., Brereton, O. P., Budgen, D., Turner, M., Bailey, J. and Linkman, S. 2009. Systematic literature reviews in software engineering: A systematic literature review. Information and Software Technology 51, 1, 7--15.

Digital Library

[14]

Kitchenham, B. A., Pretorius, R., Budgen, D., Brereton, O. P., Turner, M., Niazi, M. and Linkman, S. 2010. Systematic literature reviews in software engineering: A tertiary study. Information and Software Technology 52, 8, 792--805.

Digital Library

[15]

Greenhalgh T. and Peacock, R. 2005. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: Audit of primary sources. BMJ 331, 7524, 1064--1065.

[16]

Skoglund, M. and Runeson, P. 2009. Reference-based search strategies in systematic reviews. In Proceedings 13th Evaluation and Assessment in Software Engineering, 31-40.

Digital Library

[17]

Kitchenham, B. A., Brereton, O. P., Li, Z., Budgen, D. and Burn, A. 2011. Repeatability of systematic literature reviews. In Proceedings of the 15th International Conference on Evaluation and Assessment in Software Engineering, 46--55.

[18]

Zhang, H., Babar, M. A. and Tell, P. 2011. Identifying relevant studies in software engineering. Information and Software Technology 53, 6, 625--637.

Digital Library

[19]

Jalali, S. and Wohlin, C. 2012. Systematic literature studies: Database searches vs. backward snowballing. In Proceedings 6th International Symposium on Empirical Software Engineering and Measurement, 29--38.

Digital Library

[20]

Jalali, S. and Wohlin, C. 2012. Global software engineering and agile practices: A systematic review. Journal of Software: Evolution and Process 24, 6, 643--659.

[21]

Smite, D., Wohlin, C., Galvina, Z. and Prikladnicki, R. 2012. An empirically based terminology and taxonomy for global software engineering. Empirical Software Engineering: An International Journal 19, 1, 105--153.

Digital Library

Cited By

Santos IFelizardo KSteinmacher IGerosa M(2025)Software solutions for newcomers’ onboarding in software projects: A systematic literature reviewInformation and Software Technology10.1016/j.infsof.2024.107568177(107568)Online publication date: Jan-2025
https://doi.org/10.1016/j.infsof.2024.107568
Alhazmi AMahmud RIdris NMohamed Abo MEke C(2024)A systematic literature review of hate speech identification on Arabic Twitter data: research challenges and future directionsPeerJ Computer Science10.7717/peerj-cs.196610(e1966)Online publication date: 2-Apr-2024
https://doi.org/10.7717/peerj-cs.1966
Drolet GNiquette M(2024)De l’hostilité à la sécurité : conceptualiser la notion d’espace sécuritaire comme moyen de promotion de l’activité physique auprès des femmes considérées comme âgéesRecherches féministes10.7202/1108765ar36:1(49-68)Online publication date: 23-Jan-2024
https://doi.org/10.7202/1108765ar
Show More Cited By

Index Terms

Guidelines for snowballing in systematic literature studies and a replication in software engineering

Recommendations

Experiences from using snowballing and database searches in systematic literature studies
EASE '15: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering

Background: Systematic literature studies are commonly used in software engineering. There are two main ways of conducting the searches for these type of studies; they are snowballing and database searches. In snowballing, the reference list (backward ...
Second-generation systematic literature studies using snowballing
EASE '16: Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering

Systematic literature studies have become standard practice in software engineering to synthesize evidence in different areas of the discipline. As more such studies are published, there is also a need to extend previously published systematic ...
Systematic literature studies: database searches vs. backward snowballing
ESEM '12: Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement

Systematic studies of the literature can be done in different ways. In particular, different guidelines propose different first steps in their recommendations, e.g. start with search strings in different databases or start with the reference lists of a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

May 2014

486 pages

ISBN:9781450324762

DOI:10.1145/2601248

General Chair:
Martin Shepperd
Brunel University, UK
,
Program Chairs:
Tracy Hall
Brunel University
,
Ingunn Myrtveit
The Norwegian Business School

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Brunel University: Brunel University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Lunds Universitet

Conference

EASE '14

Sponsor:

Brunel University

EASE '14: 18th International Conference on Evaluation and Assessment in Software Engineering

May 13 - 14, 2014

England, London, United Kingdom

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1,875
Total Citations
View Citations
17,176
Total Downloads

Downloads (Last 12 months)2,980
Downloads (Last 6 weeks)241

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Santos IFelizardo KSteinmacher IGerosa M(2025)Software solutions for newcomers’ onboarding in software projects: A systematic literature reviewInformation and Software Technology10.1016/j.infsof.2024.107568177(107568)Online publication date: Jan-2025
https://doi.org/10.1016/j.infsof.2024.107568
Alhazmi AMahmud RIdris NMohamed Abo MEke C(2024)A systematic literature review of hate speech identification on Arabic Twitter data: research challenges and future directionsPeerJ Computer Science10.7717/peerj-cs.196610(e1966)Online publication date: 2-Apr-2024
https://doi.org/10.7717/peerj-cs.1966
Drolet GNiquette M(2024)De l’hostilité à la sécurité : conceptualiser la notion d’espace sécuritaire comme moyen de promotion de l’activité physique auprès des femmes considérées comme âgéesRecherches féministes10.7202/1108765ar36:1(49-68)Online publication date: 23-Jan-2024
https://doi.org/10.7202/1108765ar
Carreira FCunha PBarata JEstima J(2024)Tamper-Proof Blockchain-Based Contracts for the Carriage of Goods by RoadProceedings of the 32nd International Conference on Information Systems Development10.62036/ISD.2024.105Online publication date: 2024
https://doi.org/10.62036/ISD.2024.105
Reneilwe Maepa MNthabiseng Moeti M(2024)A Systematic Review on IoT-Based Smart Technologies for Seat Occupancy and Reservation Needs in Smart Libraries at Institution of Higher LearningInternet of Things - New Insights10.5772/intechopen.113329Online publication date: 31-Jan-2024
https://doi.org/10.5772/intechopen.113329
Teixeira RCampos WMoraes J(2024)REFLECTIONS ON THE SOCIAL CONSTRUCTION OF THE CONTEMPORARY MEDICAL PROFESSION AND CAREER: A BOURDIEUSIAN APPROACHREVISTA FOCO10.54751/revistafoco.v17n2-05517:2(e4418)Online publication date: 14-Feb-2024
https://doi.org/10.54751/revistafoco.v17n2-055
Farahbakhsh IBauch CAnand M(2024)Tipping points in coupled human–environment system models: a reviewEarth System Dynamics10.5194/esd-15-947-202415:4(947-967)Online publication date: 30-Jul-2024
https://doi.org/10.5194/esd-15-947-2024
Munisamy MOsman SSanmugam M(2024)Exploring the Landscape of Online Formative Assessment Practices in Programming CoursesIntegrating Cutting-Edge Technology Into the Classroom10.4018/979-8-3693-3124-8.ch005(75-100)Online publication date: 8-Mar-2024
https://doi.org/10.4018/979-8-3693-3124-8.ch005
Collado J(2024)Spaces of quiet resistance: navigating health and socio-spatial challenges in informal settlements during the COVID-19 pandemicTown Planning Review10.3828/tpr.2024.1795:5(527-555)Online publication date: 9-Sep-2024
https://doi.org/10.3828/tpr.2024.17
Guo HQiu HZhou YDeng Y(2024)Analysis of Influencing Factors of Level 3 Automated Vehicle Takeover: A Literature ReviewSustainability10.3390/su1619834516:19(8345)Online publication date: 25-Sep-2024
https://doi.org/10.3390/su16198345
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents