article

Combining metrics for software evolution assessment by means of Data Envelopment Analysis

Authors:

Alexander Chatzigeorgiou,

Emmanouil StiakakisAuthors Info & Claims

Journal of Software: Evolution and Process, Volume 25, Issue 3

Pages 303 - 324

https://doi.org/10.1002/smr.584

Published: 01 March 2013 Publication History

Abstract

Research and practice in software engineering have led to an extensive set of metrics for the evaluation of almost every aspect of software development. One of the major challenges for any quality model is the combination of metrics, which are complementary to each other. In this paper, we propose the use of Data Envelopment Analysis DEA, a non-parametric technique employed in economics, as a means of providing a unified view of selected design metrics. The proposed application of DEA aims at assessing the overall trend of quality during the evolution of software systems, by considering releases of a project as units to be ranked. An important benefit derived from the use of DEA is the ability to "normalize" the evaluation over the size characteristics of the examined systems, which is vital when comparing projects of different scale. Results are presented for successive versions of two open-source, one industrial and one research project, whereas validation, whereas validation is performed by comparing the findings with the results obtained by Analytic Hierarchy Process, which is an acknowledged multi-criteria decision analysis approach. According to the results, DEA enables the perception of global trends in qualitative characteristics, which would be otherwise difficult to recognize and interpret. Copyright © 2012 John Wiley & Sons, Ltd.

References

[1]

Fenton N, Pfleeger SL. Software Metrics: A Rigorous and Practical Approach. PWS Publishing Company: Boston, 1997.

Digital Library

[2]

Harrison R, Counsell SJ, Nithi RV. An evaluation of the MOOD set of object-oriented software metrics. IEEE Transactions on Software Engineering 1998; Volume 24 Issue 6: pp.491-496.

Digital Library

[3]

Subramanyam R, Krishnan MS. Empirical analysis of CK metrics for object-oriented design complexity: implications for software defects. IEEE Transactions on Software Engineering 2003; Volume 29: pp.297-310.

Digital Library

[4]

Gyimóthy T, Ferenc R, Siket I. Empirical validation of object-oriented metrics on open source software for fault prediction. IEEE Transactions on Software Engineering 2005; Volume 31 Issue 10: pp.897-910.

Digital Library

[5]

Kitchenham BA, Pickard LM, Linkman SJ. An evaluation of some design metrics. Software Engineering Journal 1990; Volume 5 Issue 1: pp.50-58.

Digital Library

[6]

Alshayeb M, Li W. An empirical validation of object-oriented metrics in two different iterative software processes. IEEE Transactions on Software Engineering 2003; Volume 29 Issue 11: pp.1043-1049.

Digital Library

[7]

Bansiya J, Davis CG. A hierarchical model for object-oriented design quality assessment. IEEE Transactions on Software Engineering 2002; Volume 28 Issue 1: pp.4-17.

Digital Library

[8]

Fenton N. New Directions for Software Metrics. CIO Symposium on Software Best Practices. Keynote Talk. September 2006. "http://www.eecs.qmul.ac.uk/~norman/papers/new_directions_for_sw_metrics.pdf" {25 January 2011}.

[9]

Kozlov D, Koskinen J, Sakkinen M, Markkula J. Assessing maintainability change over multiple software releases. Journal of Software Maintenance and Evolution: Research and Practice 2008; Volume 20 Issue 1: pp.31-58.

Digital Library

[10]

Samoladas I. Methods for Maintainability and Survival Analysis of Open-Source Software. Ph.D. Thesis, Department of Computer Science, Aristotle University of Thessaloniki, 2011.

[11]

Correia JP, Kanellopoulos Y, Visser J. A survey-based study of the mapping of system properties to ISO/IEC 9126 maintainability characteristics. 25th IEEE International Conference on Software Maintenance ICSM'2009. Edmonton, Canada. September 2009; pp.61-70.

[12]

Spinellis D, Gousios G, Karakoidas V, Louridas P, Adams PJ, Samoladas I, Stamelos I. Evaluating the quality of open source software. Electronic Notes in Theoretical Computer Science 2009; Volume 233: pp.5-28.

Digital Library

[13]

Serebrenik A, <familyNamePrefix>van den</familyNamePrefix>Brand M. Theil index for aggregation of software metrics values. 26th IEEE International Conference on Software Maintenance ICSM'2010. Timisoara, Romania. September 2010; pp.1-9.

Digital Library

[14]

Samoladas I, Stamelos I, Angelis L, Oikonomou A. Open source software development should strive for even greater code maintainability. Communications of the ACM 2004; Volume 47 Issue 10: pp.83-87.

Digital Library

[15]

Lehman MM, Ramil JF. Rules and tools for software evolution planning and management. Annals of Software Engineering 2001; Volume 11 Issue 1: pp.15-44.

Digital Library

[16]

Gîrba T, Ducasse S, Lanza M. Yesterday's weather: guiding early reverse engineering efforts by summarizing the evolution of changes. 20th IEEE International Conference on Software Maintenance ICSM'04. Chicago, IL. September 2004; pp.40-49.

Digital Library

[17]

Chatzigeorgiou A, Manakos A. Investigating the evolution of bad smells in object-oriented code. 7th International Conference on the Quality of Information and Communications Technology QUATIC'2010. Porto, Portugal. September-October 2010; pp.106-115.

Digital Library

[18]

Kagdi H, Collard ML, Maletic JI. A survey and taxonomy of approaches for mining software repositories in the context of software evolution. Journal of Software Maintenance and Evolution: Research and Practice 2007; Volume 19 Issue 2: pp.77-131.

Digital Library

[19]

Charnes A, Cooper WW, Rhodes E. Measuring the efficiency of decision making units. European Journal of Operational Research 1978; Volume 2 Issue 6: pp.429-444.

[20]

Cooper WW, Seiford LM, Tone K. Introduction to Data Envelopment Analysis and its Uses: With DEA-Solver Software and References. Springer: New York, 2005.

[21]

Chatzigeorgiou A, Stiakakis E. Benchmarking library and application software with data envelopment analysis. Software Quality Journal 2011; Volume 19 Issue 3: pp.553-578.

Digital Library

[22]

Benestad HC, Anda B, Arisholm E. Assessing software product maintainability based on class-level structural measures. 7th International Conference on Product-Focused Software Process Improvement PROFES'2006. Amsterdam, Netherlands. June 2006.

Digital Library

[23]

Oman P, Hagemeister J. Construction and testing of polynomials predicting software maintainability. Journal of Systems and Software 1994; Volume 24: pp.251-266.

Digital Library

[24]

Morisio M, Stamelos I, Tsoukias A. Software product and process assessment through profile based evaluation. International Journal of Software Engineering and Knowledge Engineering 2003; Volume 13: pp.495-512.

[25]

Samoladas I, Gousios G, Spinellis D, Stamelos I. The SQO-OSS quality model: measurement based open source software evaluation. 4th International Conference on Open Source Systems OSS'2008. Milan, Italy. September 2008; pp.237-248.

[26]

Briand L, Wüst CJ, Daly JW, Porter DV. Exploring the relationship between design measures and software quality in object-oriented systems. Journal of Systems and Software 2000; Volume 51: pp.245-273.

Digital Library

[27]

Harman M, Tratt L. Pareto optimal search based refactoring at the design level. 9th Annual Conference on Genetic and Evolutionary Computation GECCO'2007. London, UK. July 2007; pp.1106-1113.

Digital Library

[28]

O'Keeffe M, O'Cinnéide M. Search-based software maintenance. Journal of Systems and Software 2008; Volume 81 Issue 4: pp.502-516.

Digital Library

[29]

Basdavanos M, Chatzigeorgiou A. Placement of entities in object-oriented systems by means of a single-objective genetic algorithm. 5th International Conference on Software Engineering Advances ICSEA'2010. Nice, France. August 2010.

Digital Library

[30]

Bowman M, Briand L, Labiche Y. Multi-objective algorithms to support class responsibility assignment. 23rd IEEE International Conference on Software Maintenance ICSM'2007. Paris, France. October 2007; pp.124-133.

[31]

Charnes A, Cooper WW, Lewin AY, Seiford LM. Data Envelopment Analysis: Theory, Methodology, and Application. Kluwer Academic Publishers: Norwell, MA, 1996.

Digital Library

[32]

Coelli TJ, Prasada Rao DS, O'Donnell CJ, Battese GE. An Introduction to Efficiency and Productivity Analysis 2nd edn. Springer: New York, 2005.

[33]

Korpela J, Lehmusvaara A, Nisonen J. Warehouse operator selection by combining AHP and DEA methodologies. International Journal of Production Economics 2007; Volume 108: pp.135-142.

[34]

Stewart TJ. Relationships between data envelopment analysis and multicriteria decision analysis. Journal of the Operational Research Society 1996; Volume 47 Issue 5: pp.654-665.

[35]

Cooper WW, Seiford LM, Zhu J. Handbook on Data Envelopment Analysis. Kluwer Academic Publishers: New York, 2004.

[36]

Albrecht AJ. Measuring application development productivity. Proceedings of IBM Applications Development Symposium. Monterey, CA. October 1979.

[37]

Andersen P, Petersen NC. A procedure for ranking efficient units in data envelopment analysis. Management Science 1993; Volume 39: pp.1261-1264.

[38]

Javed W, McDonnel B, Elmqvist N. Graphical perception of multiple time series. IEEE Transactions on Visualization and Computer Graphics 2010; Volume 16: pp.927-934.

Digital Library

[39]

Gunopoulos D, Das G. Time series similarity measures and time series indexing. Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data SIGMOD'01. Santa Barbara, CA. May 2001.

Digital Library

[40]

Li G, Wang Y, Zhang L, Zhu X. Similarity measure for time series based on piecewise linear approximation. Proceedings of the International Conference on Wireless Communications & Signaling Processing WCSP 2009. Nanjing, China. November 2009; pp.1-4.

[41]

Holzhüter C, Hadlak S, Schumann H. Multi-level visualization for the exploration of temporal trends in simulation data. Winter Simulation Conference WSC 2010. Baltimore, MD. December 2010.

[42]

Saaty T. A scaling method for priorities in hierarchical structures. Journal of Mathematical Psychology 1977; Volume 15 Issue 3: pp.234-281.

[43]

Forman EH, Gass SI. The Analytic Hierarchy Process - an exposition. Operations Research 2001; Volume 49 Issue 4: pp.469-486.

Digital Library

[44]

Wang YM, Elhag TMS. An approach to avoiding rank reversal in AHP. Decision Support Systems 2006; Volume 42 Issue 3: pp.1474-1480.

Digital Library

[45]

Forman EH. Facts and fictions about the Analytic Hierarchy Process. Mathematical and Computer Modelling 1993; Volume 17 Issue 4-5: pp.19-26.

Digital Library

[46]

Ahmad N, Laplante PA. Employing expert opinion and software metrics for reasoning about software. 3rd IEEE International Symposium on Dependable, Autonomic and Secure Computing DASC'2007. Columbia, MD. September 2007; pp.119-124.

Digital Library

[47]

Li W, Henry S. Object-oriented metrics that predict maintainability. Journal of Systems and Software 1993; Volume 23 Issue 2: pp.111-122.

Digital Library

[48]

Henderson-Sellers B, Constantine LL, Graham IM. Coupling and cohesion towards a valid metrics suite for object-oriented analysis and design. Object-Oriented Systems 1996; Volume 3 Issue 3: pp.143-158.

[49]

Briand LC, Daly J, Porter V, Wüst J. A comprehensive empirical validation of design measures for object-oriented systems. Proceedings of the 5th International Symposium on Software Metrics METRICS'98. Bethesda, MD. March 1998; pp.246-257.

Digital Library

[50]

Briand L, Daly J, Wüst J. A unified framework for cohesion measurement in object-oriented systems. Empirical Software Engineering Journal 1998; Volume 3 Issue 1: pp.65-117.

Digital Library

[51]

McCabe TJ. A complexity measure. IEEE Transactions on Software Engineering 1976; Volume SE-2 Issue 4: pp.308-320.

Digital Library

[52]

Lee Y, Yang J, Chang KH. Metrics and evolution in open source software. 7th International Conference on Quality Software QSIC'2007. Portland, OR. October 2007; pp.191-197.

Digital Library

[53]

International Organization for Standardization/International Electrotechnical Commission. ISO/IEC 9126 - Software Engineering - Product Quality, Geneve, 2001.

[54]

Chidamber SR, Kemerer CF. A metrics suite for object oriented design. IEEE Transactions on Software Engineering 1994; Volume 20 Issue 6: pp.476-493.

Digital Library

[55]

Waldo S. Efficiency in public education. Nationalekonomiska Institutionen, Lund University, Working Paper Series, 2002. "http://www.nek.lu.se/publications/workpap/Papers/WP02_10.pdf" {27 May 2011}.

[56]

Olesen O, Petersen N. Incorporating quality into data envelopment analysis: a stochastic dominance approach. International Journal of Production Economics 1995; Volume 39: pp.117-135.

[57]

DEA_AHP Material. "http://java.uom.gr/OOD/DEA_AHP_Material.pdf" {2 October 2011}.

[58]

SAITECH Inc. - Products - DEA Solver PRO. "http://www.saitech-inc.com/Products/Prod-DSP.asp" {20 September 2010}.

[59]

Software Architecture Design. Visual UML & Business Process Modeling. "http://www.borland.com/us/products/together/index.aspx" {20 September 2010}.

[60]

Robust Decision-Support and Analysis Platform - Expert Choice. "http://www.expertchoice.com/products-services/expert-choice-115" {20 September 2010}.

[61]

Bar-Ilan J, Mat-Hassan M, Levene M. Methods for comparing rankings of search engine results. Computer Networks 2006; Volume 50 Issue 10: pp.1448-1463.

Digital Library

[62]

Bar-Ilan J. Rankings of information and library science journals by JIF and by h-type indices. Journal of Informetrics 2010; Volume 4: pp.141-147.

[63]

Understand your Code. "http://www.scitools.com/" {15 May 2011}.

[64]

Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A. Experimentation in Software Engineering: An Introduction. Kluwer: Boston, MA, 2000.

Digital Library

Cited By

Arvanitou EAmpatzoglou AChatzigeorgiou AAvgeriou P(2018)Software metrics fluctuationInformation and Software Technology10.1016/j.infsof.2015.12.01072:C(110-124)Online publication date: 30-Dec-2018
https://dl.acm.org/doi/10.1016/j.infsof.2015.12.010
Przybyłek A(2018)An empirical study on the impact of AspectJ on software evolvabilityEmpirical Software Engineering10.1007/s10664-017-9580-723:4(2018-2050)Online publication date: 1-Aug-2018
https://dl.acm.org/doi/10.1007/s10664-017-9580-7
Feitosa DAmpatzoglou AAvgeriou PNakagawa EKruchten POzkaya IKoziolek H(2015)Investigating Quality Trade-offs in Open Source Critical Embedded SystemsProceedings of the 11th International ACM SIGSOFT Conference on Quality of Software Architectures10.1145/2737182.2737190(113-122)Online publication date: 4-May-2015
https://dl.acm.org/doi/10.1145/2737182.2737190
Show More Cited By

Combining metrics for software evolution assessment by means of Data Envelopment Analysis

Recommendations

Guidelines for the Application of Data Envelopment Analysis to Assess Evolving Software
Revised Selected Papers of the SEFM 2012 Satellite Events on Information Technology and Open Source: Applications for Education, Innovation, and Sustainability - Volume 7991

The assessment of software evolution in terms of quality poses significant challenges as different metrics have to be combined and normalized over the size of each examined version. Data Envelopment Analysis DEA, a non-parametric technique from ...
Detection of Software Anomalies Using Object-oriented Metrics
ICEIS 2014: Proceedings of the 16th International Conference on Enterprise Information Systems - Volume 2

The development of quality software has always been the aim of many studies in past years, in which the

focus was on seeking for better software production with high effectiveness and quality. In order to evaluate software quality, software metrics were ...
An Evolution of Software Metrics: A Review
ICAIP '17: Proceedings of the International Conference on Advances in Image Processing

The main goal of software engineering is to produce good quality reusable software within the given time-period, with minimum cost and that satisfies the user's needs. To enhance the quality of the software, number of techniques should be adopted. In ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Journal of Software: Evolution and Process

Journal of Software: Evolution and Process Volume 25, Issue 3

March 2013

130 pages

ISSN:2047-7473

EISSN:2047-7481

Issue’s Table of Contents

Publisher

John Wiley & Sons, Inc.

United States

Publication History

Published: 01 March 2013

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Arvanitou EAmpatzoglou AChatzigeorgiou AAvgeriou P(2018)Software metrics fluctuationInformation and Software Technology10.1016/j.infsof.2015.12.01072:C(110-124)Online publication date: 30-Dec-2018
https://dl.acm.org/doi/10.1016/j.infsof.2015.12.010
Przybyłek A(2018)An empirical study on the impact of AspectJ on software evolvabilityEmpirical Software Engineering10.1007/s10664-017-9580-723:4(2018-2050)Online publication date: 1-Aug-2018
https://dl.acm.org/doi/10.1007/s10664-017-9580-7
Feitosa DAmpatzoglou AAvgeriou PNakagawa EKruchten POzkaya IKoziolek H(2015)Investigating Quality Trade-offs in Open Source Critical Embedded SystemsProceedings of the 11th International ACM SIGSOFT Conference on Quality of Software Architectures10.1145/2737182.2737190(113-122)Online publication date: 4-May-2015
https://dl.acm.org/doi/10.1145/2737182.2737190
Paschalidou GStiakakis EChatzigeorgiou ADiamantaras KEvangelidis GManolopoulos YGeorgiadis CKefalas PStamatis D(2013)An application of data envelopment analysis to software quality assessmentProceedings of the 6th Balkan Conference in Informatics10.1145/2490257.2490264(228-235)Online publication date: 19-Sep-2013
https://dl.acm.org/doi/10.1145/2490257.2490264
Chatzigeorgiou A(2012)Guidelines for the Application of Data Envelopment Analysis to Assess Evolving SoftwareRevised Selected Papers of the SEFM 2012 Satellite Events on Information Technology and Open Source: Applications for Education, Innovation, and Sustainability - Volume 799110.1007/978-3-642-54338-8_23(281-287)Online publication date: 1-Oct-2012
https://dl.acm.org/doi/10.1007/978-3-642-54338-8_23

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents