An Analysis of the Empirical Software Engineering over the last 10 Editions of Brazilian Software Engineering Symposium
Authors: 
Davi Monteiro, Rômulo Gadelha, Thayse Alencar, Bruno Neves, Italo Yeltsin, Thiago Gomes, Mariela CortésAuthors Info & Claims
Pages 44 - 53
Abstract
Empirical evaluations developed in the software engineering area have been widely applied as a formalism to validate and ensure the credibility of the works proposed by the researchers. Even though the adoption of empirical evaluation techniques has gained popularity in recent years, its application has been questioned both qualitatively and quantitatively. This study aims at analyzing how empirical software engineering research has evolved in the Brazilian Symposium on Software Engineering (SBES) community. We performed a controlled quasi-experiment, using published papers over the last 10 years in SBES. Our experiment was divided into two phases: classification by type and quality assessment of the main empirical types. In the first phase, the sample was 201 papers; in the second one, the sample decreased to 126 papers. The results have shown failures and gaps in the application of empirical methods when assessing the quality of the Software Engineering works. We believe that we can contribute to improve how the studies were conducted and consequently help to produce more reliable results, reducing or eliminating biases: an important qualitative factor in scientific work. In addition, due to the lack of assessment supporting tools, we developed a theoretical protocol to support the assessment process and proposed improvements for papers that obtained below-expected rates.
References
[1]
Victor R. Basili. 1996. The Role of Experimentation in Software Engineering: Past, Current, and Future. In Proceedings of the 18th International Conference on Software Engineering (ICSE). IEEE Computer Society, 442--449.
[2]
Jeffrey C. Carver, Eugene Syriani, and Jeff Gray. 2011. Assessing the Frequency of Empirical Evaluation in Software Modeling Research. In Proceedings of the First International Workshop on Experiences and Empirical Studies in Software Modelling. CEUR-WS.org, 28--37.
[3]
A. Fink. 2003. The Survey Handbook. SAGE Publications.
[4]
G.V. Glass, B. McGaw, and M.L. Smith. 1981. Meta-analysis in social research. Sage Publications.
[5]
Staffs Keele. 2007. Guidelines for performing systematic literature reviews in software engineering. In Technical report, Ver. 2.3 EBSE Technical Report. EBSE. sn.
[6]
Barbara Kitchenham, O. Pearl Brereton, David Budgen, Mark Turner, John Bailey, and Stephen Linkman. 2009. Systematic Literature Reviews in Software Engineering - A Systematic Literature Review. Information and Software Technology 51, 1 (Jan. 2009), 7--15.
[7]
Barbara A. Kitchenham, O. Pearl Brereton, David Budgen, and Zhi Li. 2009. An Evaluation of Quality Checklist Proposals: A Participant-observer Case Study. In Proceedings of the 13th International Conference on Evaluation and Assessment in Software Engineering (EASE'09). BCS Learning & Development Ltd., Swindon, UK, 55--64. http://dl.acm.org/citation.cfm?id=2227040.2227047
[8]
Barbara Ann Kitchenham, David Budgen, and Pearl Brereton. 2015. Evidence-Based Software Engineering and Systematic Reviews. Chapman & Hall/CRC.
[9]
Barbara A Kitchenham, Tore Dyba, and Magne Jorgensen. 2004. Evidence-based software engineering. In Proceedings of the 26th international conference on software engineering. IEEE Computer Society, 273--281.
[10]
Robert V. Labaree. 2017. Research Methods in the Social Sciences. (June 2017). http://lynn-library.libguides.com/c.php?g=549455&p=3771805
[11]
Johan Linåker, Sardar Muhammad Sulaman, Rafael Maiani de Mello, and Martin Höst. 2015. Guidelines for conducting surveys in software engineering. http://portal.research.lu.se/portal/files/6062997/5463412.pdf. (2015). Accessed 10 December 2016.
[12]
Ruchika Malhotra. 2015. Empirical Research in Software Engineering: Concepts, Analysis, and Applications. Chapman & Hall/CRC.
[13]
Marcus R Munafò, Brian A Nosek, Dorothy VM Bishop, Katherine S Button, Christopher D Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J Ware, and John PA Ioannidis. 2017. A manifesto for reproducible science. Nature Human Behaviour 1 (2017), 0021.
[14]
Dewayne E. Perry, Susan Elliott Sim, and Steve M. Easterbrook. 2004. Case Studies for Software Engineers. In Proceedings of the 26th International Conference on Software Engineering (ICSE '04). IEEE Computer Society, Washington, DC, USA, 736--738.
[15]
Per Runeson and Martin Höst. 2009. Guidelines for Conducting and Reporting Case Study Research in Software Engineering. Empirical Software Engineering 14, 2 (April 2009), 131--164.
[16]
Per Runeson, Martin Host, Austen Rainer, and Bjorn Regnell. 2012. Case Study Research in Software Engineering: Guidelines and Examples. Wiley Publishing.
[17]
Mary Shaw. 2003. Writing good software engineering research papers. In Proceedings of the 25th International Conference on Software Engineering (ICSE). IEEE, 726--736.
[18]
Paulo Anselmo Da Mota Silveira Neto, JoáS Sousa Gomes, Eduardo Santana De Almeida, Jair Cavalcanti Leite, Thais Vasconcelos Batista, and Larissa Leite. 2013. 25 Years of Software Engineering in Brazil: Beyond an Insider's View. J. Syst. Softw. 86, 4 (April 2013), 872--889.
[19]
Dag I. K. Sjoberg, Tore Dyba, and Magne Jorgensen. 2007. The Future of Empirical Methods in Software Engineering Research. In Proceedings of the Future of Software Engineering (FOSE). IEEE Computer Society, 358--378.
[20]
Andrs Vargha and Harold D. Delaney. 2000. A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics 25, 2 (2000), 101--132.
[21]
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.
[22]
R.K. Yin. 2009. Case Study Research: Design and Methods. SAGE Publications.
[23]
Carmen Zannier, Grigori Melnik, and Frank Maurer. 2006. On the Success of Empirical Studies in the International Conference on Software Engineering. In Proceedings of the 28th International Conference on Software Engineering (ICSE '06). ACM, New York, NY, USA, 341--350.
Index Terms
- An Analysis of the Empirical Software Engineering over the last 10 Editions of Brazilian Software Engineering Symposium
Recommendations
Issues in applying empirical software engineering to software architecture
ECSA'07: Proceedings of the First European conference on Software ArchitectureEmpirical software engineering focuses on the evaluation of software engineering technologies, such as processes and tools, by comparing related sets of data. It has contributed a valuable body of knowledge in several areas such as Software Economics ...
Empirical software engineering for agent programming
AGERE! 2012: Proceedings of the 2nd edition on Programming systems, languages and applications based on actors, agents, and decentralized control abstractionsEmpirical software engineering is a branch of software engineering in which empirical methods are used to evaluate and develop tools, languages and techniques. In this position paper we argue for the use of empirical methods to advance the area of agent ...
Applying empirical software engineering to software architecture: challenges and lessons learned
In the last 15 years, software architecture has emerged as an important software engineering field for managing the development and maintenance of large, software-intensive systems. Software architecture community has developed numerous methods, ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
September 2017
409 pages
ISBN:9781450353267
DOI:10.1145/3131151
- Program Chairs:
- José Carlos Maldonado,
- Fabiano Cutigi Ferrari,
- Uirá Kulesza,
- Tayana Uchôa Conte
Copyright © 2017 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
In-Cooperation
- SBC: Brazilian Computer Society
- CNPq: Conselho Nacional de Desenvolvimento Cientifico e Tecn
- CAPES: Brazilian Higher Education Funding Council
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 20 September 2017
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
SBES'17
SBES'17: 31st Brazilian Symposium on Software Engineering
September 20 - 22, 2017
CE, Fortaleza, Brazil
Acceptance Rates
SBES '17 Paper Acceptance Rate 42 of 134 submissions, 31%;
Overall Acceptance Rate 147 of 427 submissions, 34%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 114Total Downloads
- Downloads (Last 12 months)7
- Downloads (Last 6 weeks)1
Reflects downloads up to 20 Nov 2024
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in