Replication of Empirical Studies in Software Engineering: Preliminary Findings from a Systematic Mapping Study

Our goal in this study is to review the research related to the replication of empirical studies in software engineering in terms of replications of empirical studies and conceptual or theoretical work about replications. In this article we present the preliminary findings of this review, concentrating on the studies reporting replications and the related original studies. We applied the systematic review method to perform a mapping study about the current state of the replication work of empirical studies performed in software engineering research. We analyzed 16,126 articles, from which we extracted 93 articles reporting 125 replications performed between 1994 and 2010, of 76 original studies. Over 60% of the replications were performed in the last six years and 71% percent of the studies were internal replications. The topics of software construction, testing, and maintenance concentrate nearly 50% of the replication work, while software design, configuration management and software tools and methods are the topics with least replications. The number of replications grew in the last few years, but the absolute number of replications is still very small, in particular considering the breadth of topics in software engineering. Incentive to perform external replications and better standards to report empirical studies and their replications are still needed.