article

Semi-automatic selection of primary studies in systematic literature reviews: is it reasonable?

Authors:

Fábio R. Octaviano,

Katia R. Felizardo,

José C. Maldonado,

Sandra C. FabbriAuthors Info & Claims

Empirical Software Engineering, Volume 20, Issue 6

Pages 1898 - 1917

https://doi.org/10.1007/s10664-014-9342-8

Published: 01 December 2015 Publication History

Abstract

The systematic review (SR) is a methodology used to find and aggregate all relevant existing evidence about a specific research question of interest. One of the activities associated with the SR process is the selection of primary studies, which is a time consuming manual task. The quality of primary study selection impacts the overall quality of SR. The goal of this paper is to propose a strategy named "Score Citation Automatic Selection" (SCAS), to automate part of the primary study selection activity. The SCAS strategy combines two different features, content and citation relationships between the studies, to make the selection activity as automated as possible. Aiming to evaluate the feasibility of our strategy, we conducted an exploratory case study to compare the accuracy of selecting primary studies manually and using the SCAS strategy. The case study shows that for three SRs published in the literature and previously conducted in a manual implementation, the average effort reduction was 58.2 % when applying the SCAS strategy to automate part of the initial selection of primary studies, and the percentage error was 12.98 %. Our case study provided confidence in our strategy, and suggested that it can reduce the effort required to select the primary studies without adversely affecting the overall results of SR.

References

[1]

Babar MA, Zhang H (2010) Systematic literature reviews in software engineering: Preliminary results from interviews with researchers. In: Proc, ESEM'09, 1-10.

[2]

Boell S, Cezec-Kecmanovic D (2011) "Are systematic reviews better, less biased and of higher quality?" ECIS 2011 Proceedings. Paper 223. http://aisel.aisnet.org/ecis2011/223.

[3]

Brereton OP, Kitchenham BA, Budgen D, Turner M, Khalil M (2007) Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw 80(1):571-583.

Digital Library

[4]

Carletta J (1996) Assessing agreement on classification tasks: the kappa statistic. Comput Linguis 22(2):249-254.

Digital Library

[5]

Dieste O, Grimán A, Juristo N (2009) Developing search strategies for detecting relevant experiments. In: Proc. ESEM'09, 513-539.

[6]

Dybå T, Kitchenham BA, Jorgensen M (2005) Evidence-based software engineering for practitioners. IEEE Softw 22(1):58-65.

Digital Library

[7]

Dybå T, Dingsøyr T, Hanssen GK (2007) Applying systematic reviews to diverse study types: An experience report. In: Proc. ESEM'07, 225-234.

[8]

Fabbri S, Hernandes E, Di Thommazo A, Belgamo A, Zamboni A, Silva C (2012) Managing literature reviews information through visualization. In: Proc. ICEIS'12, 36-45.

[9]

Felizardo KR, Salleh N, Martins R, Mendes E, MacDonell S, Maldonado JC (2011) Using visual text mining to support the study selection activity in systematic literature reviews. In: Proc. ESEM'11, 1-10.

[10]

Felizardo KR, Andery G, Paulovich F, Minghim R, Maldonado JC (2012) Avisual analysis approach to validate the selection review of primary studies in systematic reviews. Inf Softw Technol 54(10):1079-1091.

Digital Library

[11]

Kitchenham BA, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. EBSE Technical Report, Keele University and University of Durham, version 2.3.

[12]

Kitchenham BA, Dybå T, Jørgensen M (2004) Evidence-based software engineering. In: Proc. ICSE'08, 273-281.

[13]

Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159- 174.

[14]

Malheiros V, Hohn E, Pinho R, Mendonca M, Maldonado JC (2007) A visual text mining approach for systematic reviews. In: Proc. ESEM'07, 245-254.

[15]

Marshall C, Brereton P (2013) Tools to Support Systematic Literature Reviews in Software Engineering: A Mapping Study. In: Proc. ESEM'13, 296-299.

[16]

Oates BJ, Capper G (2009) Using systematic reviews and evidence-based software engineering with masters students. In: Proc. EASE'09, 1-8.

[17]

Paulovich FV, Minghim R (2008) HiPP: a novel hierarchical point placement strategy and its application to the exploration of document collections. IEEE Trans Vis Comput Graph 14(6):1229-1236.

[18]

Petersen K, Ali NB (2011) Identifying strategies for study selection in systematic reviews and maps. In: Proc. ESEM'11, 351-354.

[19]

Riaz M, Sulayman M, Salleh N, Mendes E (2010) Experiences conducting systematic reviews from novices' perspective. In: Proc. EASE'10, 1-10.

[20]

Shaw M, Clements P (2006) The golden age of software architecture: A comprehensive survey. Technical Report CMU-ISRI-06-101, Software Engineering Institute, Carnegie Mellon University.

[21]

Shepperd M (2007) Software project economics: A roadmap. In: Proc. FOSE'07, IEEE Computer Society, Washington, DC, pp 304-315.

[22]

Weka (2013) Available at http://www.cs.waikato.ac.nz/ml/weka/index.html. Last access on 30-March- 2013.

[23]

Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2000) Experimentation in Software Engineering: An Introduction. In: Kluwer Academic Publishers (ed), 1st edn. Boston, US, pp 63-74.

[24]

Zhang H, Muhammad AB (2011) An empirical investigation of systematic reviews in software engineering. In: Proc. ESEM'11, 1-10.

Cited By

de la Torre-López JRamírez ARomero J(2023)Artificial intelligence to automate the systematic review of scientific literatureComputing10.1007/s00607-023-01181-x105:10(2171-2194)Online publication date: 11-May-2023
https://dl.acm.org/doi/10.1007/s00607-023-01181-x
Shakeel YKrüger JNostitz-Wallwitz ISaake GLeich T(2019)Automated Selection and Quality Assessment of Primary StudiesJournal of Data and Information Quality10.1145/335690112:1(1-26)Online publication date: 16-Nov-2019
https://dl.acm.org/doi/10.1145/3356901
de Oliveira Neto FTorkar RFeldt RGren LFuria CHuang Z(2019)Evolution of statistical analysis in empirical software engineering researchJournal of Systems and Software10.1016/j.jss.2019.07.002156:C(246-267)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1016/j.jss.2019.07.002
Show More Cited By

Recommendations

Using the SCAS strategy to perform the initial selection of studies in systematic reviews: an experimental study
EASE '16: Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering

Context: Systematic Review (SR) is a well-defined and rigorous methodology used to find relevant evidence about a specific topic of interest. Depending on the number of identified primary studies, the selection activity can be very time-consuming and a ...
Using Visual Text Mining to Support the Study Selection Activity in Systematic Literature Reviews
ESEM '11: Proceedings of the 2011 International Symposium on Empirical Software Engineering and Measurement

Background: A systematic literature review (SLR) is a methodology used to aggregate all relevant existing evidence to answer a research question of interest. Although crucial, the process used to select primary studies can be arduous, time consuming, ...
Dynamic Channel Selection for Cognitive Radios with Heterogenous Primary Bands

Technological advances coupled with regulatory initiative for more efficient utilization of radio spectrum resulted in the introduction of dynamic spectrum access enabled radio called Cognitive Radio (CR). A CR network is allowed to access a frequency ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Empirical Software Engineering

Empirical Software Engineering Volume 20, Issue 6

December 2015

491 pages

ISSN:1382-3256

Issue’s Table of Contents

Copyright © Copyright © 2015 Springer Science+Business Media New York.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 December 2015

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

de la Torre-López JRamírez ARomero J(2023)Artificial intelligence to automate the systematic review of scientific literatureComputing10.1007/s00607-023-01181-x105:10(2171-2194)Online publication date: 11-May-2023
https://dl.acm.org/doi/10.1007/s00607-023-01181-x
Shakeel YKrüger JNostitz-Wallwitz ISaake GLeich T(2019)Automated Selection and Quality Assessment of Primary StudiesJournal of Data and Information Quality10.1145/335690112:1(1-26)Online publication date: 16-Nov-2019
https://dl.acm.org/doi/10.1145/3356901
de Oliveira Neto FTorkar RFeldt RGren LFuria CHuang Z(2019)Evolution of statistical analysis in empirical software engineering researchJournal of Systems and Software10.1016/j.jss.2019.07.002156:C(246-267)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1016/j.jss.2019.07.002
Mannocci AOsborne FMotta E(2019)The evolution of IJHCS and CHIInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2019.05.009131:C(23-40)Online publication date: 1-Nov-2019
https://dl.acm.org/doi/10.1016/j.ijhcs.2019.05.009
Shakeel YKrüger Jvon Nostitz-Wallwitz ILausberger CDurand GSaake GLeich TCarver JHong NJay C(2018)(Automated) literature analysisProceedings of the International Workshop on Software Engineering for Science10.1145/3194747.3194748(20-27)Online publication date: 2-Jun-2018
https://dl.acm.org/doi/10.1145/3194747.3194748
Fabbri SSilva CHernandes EOctaviano FDi Thommazo ABelgamo ABeecham SKitchenham BMacDonell S(2016)Improvements in the StArt tool to better support the systematic review processProceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering10.1145/2915970.2916013(1-5)Online publication date: 1-Jun-2016
https://dl.acm.org/doi/10.1145/2915970.2916013
Octaviano FSilva CFabbri SBeecham SKitchenham BMacDonell S(2016)Using the SCAS strategy to perform the initial selection of studies in systematic reviewsProceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering10.1145/2915970.2916000(1-10)Online publication date: 1-Jun-2016
https://dl.acm.org/doi/10.1145/2915970.2916000

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents