research-article

Evaluating usefulness of software metrics: an industrial experience report

Authors:

Arie van Deursen,

Joost VisserAuthors Info & Claims

ICSE '13: Proceedings of the 2013 International Conference on Software Engineering

Pages 921 - 930

Published: 18 May 2013 Publication History

Abstract

A wide range of software metrics targeting various abstraction levels and quality attributes have been proposed by the research community. For many of these metrics the evaluation consists of verifying the mathematical properties of the metric, investigating the behavior of the metric for a number of open-source systems or comparing the value of the metric against other metrics quantifying related quality attributes. Unfortunately, a structural analysis of the usefulness of metrics in a real-world evaluation setting is often missing. Such an evaluation is important to understand the situations in which a metric can be applied, to identify areas of possible improvements, to explore general problems detected by the metrics and to define generally applicable solution strategies. In this paper we execute such an analysis for two architecture level metrics, Component Balance and Dependency Profiles, by analyzing the challenges involved in applying these metrics in an industrial setting. In addition, we explore the usefulness of the metrics by conducting semi-structured interviews with experienced assessors. We document the lessons learned both for the application of these specific metrics, as well as for the method of evaluating metrics in practice.

References

[1]

A. Abran. Software Metrics and Software Metrology. Wiley-IEEE Computer Society Pr, 2010.

Digital Library

[2]

S. Adolph, W. Hall, and P. Kruchten. Using grounded theory to study the experience of software development. Empirical Software Engineering, 16(4):487–513, Aug. 2011.

Digital Library

[3]

A. Albrecht and J. Gaffney, J.E. Software function, source lines of code, and development effort prediction: A software science validation. Software Engineering, IEEE Transactions on, SE-9(6):639 – 648, 1983.

Digital Library

[4]

T. L. Alves, J. P. Correia, and J. Visser. Benchmark-based aggregation of metrics to ratings. In IWSM/Mensura, pages 20–29, 2011.

Digital Library

[5]

R. Baggen, K. Schill, and J. Visser. Standardized code quality benchmarking for improving software maintainability. In 4th International Workshop on Software Quality and Maintainability (SQM 2010), 2010.

[6]

V. R. Basili, G. Caldiera, and H. D. Rombach. The goal question metric approach. In Encyclopedia of Software Engineering. Wiley, 1994.

[7]

E. Bouwers, J. Correia, A. van Deursen, and J. Visser. Quantifying the analyzability of software architectures. In Proceedings of the 9th Working IEEE/IFIP Conference on Software Architecture (WICSA 2011). IEEE Computer Society, 2011.

Digital Library

[8]

E. Bouwers, A. van Deursen, and J. Visser. Dependency profiles for software architecture evaluations. In Proceedings of the 27th IEEE International Conference on Software Maintenance (ICSM 2011). IEEE Computer Society, 2011.

Digital Library

[9]

E. Bouwers, A. van Deursen, and J. Visser. Quantifying the encapsulation of implemented software architectures. Technical Report TUDSERG-2011-031, Delft Software Engineering Research Group, Delft University of Technology, 2011.

[10]

L. C. Briand, J. W. Daly, and J. Wüst. A unified framework for cohesion measurement in object-orientedsystems. Empirical Software Engineering, 3(1):65–117, July 1998.

Digital Library

[11]

L. C. Briand, S. Morasca, and V. R. Basili. Defining and validating measures for object-based high-level design. IEEE Transactions on Software Engineering, 25(5):722–743, 1999.

Digital Library

[12]

S. Chidamber and C. Kemerer. A metrics suite for object oriented design. IEEE Transactions on Software Engineering, 20:476–493, 1994.

Digital Library

[13]

F. D. Davis. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q., 13(3):319–340, 1989.

Digital Library

[14]

A. v. Deursen and T. Kuipers. Source-based software risk assessment. In ICSM ’03: Proceedings of the International Conference on Software Maintenance. IEEE Computer Society, 2003.

Digital Library

[15]

N. E. Fenton and S. L. Pfleeger. Software Metrics: A Rigorous and Practical Approach. PWS Publishing Co., Boston, MA, USA, 2nd edition, 1998.

Digital Library

[16]

A. Gopal, T. Mukhopadhyay, and M. Krishnan. The impact of institutional forces on software metrics programs. Software Engineering, IEEE Transactions on, 31(8):679 – 694, aug. 2005.

Digital Library

[17]

M. Greiler, A. van Deursen, and M. Storey. Test confessions: a study of testing practices for plug-in systems. In Proceedings of the 2012 International Conference on Software Engineering, ICSE 2012, pages 244–254, Piscataway, NJ, USA, 2012. IEEE Press.

Digital Library

[18]

I. Heitlager, T. Kuipers, and J. Visser. A practical model for measuring maintainability. In QUATIC ’07: Proc. 6th Int. Conf. on Quality of Information and Communications Technology, pages 30–39. IEEE Computer Society, 2007.

Digital Library

[19]

International Organization for Standardization. ISO/IEC 9126-1: Software engineering - product quality - part 1: Quality model, 2001.

[20]

International Organization for Standardization. ISO/IEC 25010: Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models, 2011.

[21]

C. Kaner and W. P. Bond. Software engineering metrics: What do they measure and how do we know? In 10TH International Software Metrics Symposium - Metrics 2004, 2004.

[22]

B. Kitchenham. Whats up with software metrics? A preliminary mapping study. Journal of Systems and Software, 83(1):37 – 51, 2010.

Digital Library

[23]

H. Koziolek. Sustainability evaluation of software architectures: a systematic review. In Proceedings of the joint ACM SIGSOFT conference – QoSA and ACM SIGSOFT symposium – ISARCS on Quality of software architectures – QoSA and architecting critical systems – ISARCS, QoSAISARCS ’11, pages 3–12, New York, NY, USA, 2011. ACM.

Digital Library

[24]

T. Kuipers and J. Visser. A tool-based methodology for software portfolio monitoring. In Software Audit and Metrics, Proceedings of the 1st International Workshop on Software Audit and Metrics, pages 118–128. INSTICC Press., 2004.

[25]

H. Lu, Y. Zhou, B. Xu, H. Leung, and L. Chen. The ability of objectoriented metrics to predict change-proneness: a meta-analysis. Empirical Software Engineering, 17:200–242, 2012. 10.1007/s10664-011-9170-z.

Digital Library

[26]

T. J. McCabe. A complexity measure. In ICSE ’76: Proceedings of the 2nd international conference on Software engineering. IEEE Computer Society Press, 1976.

Digital Library

[27]

A. Meneely, B. Smith, and L. Williams. Validating software metrics: A spectrum of philosophies. ACM Transactions on Software Engineering and Methodology (TOSEM), 21, 2012.

Digital Library

[28]

N. Nagappan and T. Ball. Static analysis tools as early indicators of pre-release defect density. In Proceedings of the 27th international conference on Software engineering, ICSE ’05, pages 580–586, New York, NY, USA, 2005. ACM.

Digital Library

[29]

S. Sarkar, A. C. Kak, and G. M. Rama. Metrics for measuring the quality of modularization of large-scale object-oriented software. IEEE Transactions on Software Engineering, 34:700–720, 2008.

Digital Library

[30]

S. Sarkar, G. M. Rama, and A. C. Kak. API-based and informationtheoretic metrics for measuring the quality of software modularization. IEEE Transactions of Software Engineering, 33(1):14–32, 2007.

Digital Library

Cited By

Barreto Simedo Pacheco LRahman MRabbi FFathollahzadeh PAbdellatif AShihab EChen TYang JZou YBosch JLewis GCleland-Huang JMuccini H(2024)DVC in Open Source ML-development: The Action and the ReactionProceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI10.1145/3644815.3644965(75-80)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3644815.3644965
Zhou SVasilescu BKästner CDumas MPfahl DApel SRusso A(2019)What the fork: a study of inefficient and efficient forking practices in social codingProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338918(350-361)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338918
di Biase MRastogi ABruntink Mvan Deursen AAvgeriou PSchmid K(2019)The delta maintainability modelProceedings of the Second International Conference on Technical Debt10.1109/TechDebt.2019.00030(113-122)Online publication date: 26-May-2019
https://dl.acm.org/doi/10.1109/TechDebt.2019.00030
Show More Cited By

Index Terms

Evaluating usefulness of software metrics: an industrial experience report

Recommendations

Comparing software metrics tools
ISSTA '08: Proceedings of the 2008 international symposium on Software testing and analysis

This paper shows that existing software metric tools interpret and implement the definitions of object-oriented software metrics differently. This delivers tool-dependent metrics results and has even implications on the results of analyses based on ...
Evaluating and comparing software metrics in the software engineering laboratory

There has appeared in the literature a great number of metrics that attempt to measure the effort or complexity in developing and understanding software(1). There have also been several attempts to independently validate these measures on data from ...
Evaluating and comparing software metrics in the software engineering laboratory
Proceedings of the 1981 ACM workshop/symposium on Measurement and evaluation of software quality

There has appeared in the literature a great number of metrics that attempt to measure the effort or complexity in developing and understanding software(1). There have also been several attempts to independently validate these measures on data from ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '13: Proceedings of the 2013 International Conference on Software Engineering

May 2013

1561 pages

ISBN:9781467330763

General Chair:
David Notkin,
Program Chairs:
Betty H. C. Cheng,
Klaus Pohl

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

IEEE Press

Publication History

Published: 18 May 2013

Check for updates

Qualifiers

Research-article

Conference

ICSE '13

Sponsor:

SIGSOFT

ICSE '13: 35th International Conference on Software Engineering

May 18 - 26, 2013

CA, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
458
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Barreto Simedo Pacheco LRahman MRabbi FFathollahzadeh PAbdellatif AShihab EChen TYang JZou YBosch JLewis GCleland-Huang JMuccini H(2024)DVC in Open Source ML-development: The Action and the ReactionProceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI10.1145/3644815.3644965(75-80)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3644815.3644965
Zhou SVasilescu BKästner CDumas MPfahl DApel SRusso A(2019)What the fork: a study of inefficient and efficient forking practices in social codingProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338918(350-361)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338918
di Biase MRastogi ABruntink Mvan Deursen AAvgeriou PSchmid K(2019)The delta maintainability modelProceedings of the Second International Conference on Technical Debt10.1109/TechDebt.2019.00030(113-122)Online publication date: 26-May-2019
https://dl.acm.org/doi/10.1109/TechDebt.2019.00030
Mo RSnipes WCai YRamaswamy SKazman RNaedele MHuchard MKästner CFraser G(2018)Experiences applying automated architecture analysis tool suitesProceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering10.1145/3238147.3240467(779-789)Online publication date: 3-Sep-2018
https://dl.acm.org/doi/10.1145/3238147.3240467
Le DBehnamghader PGarcia JLink DShahbazian AMedvidovic NPenta MPinzger MRobbes R(2015)An empirical study of architectural change in open-source software systemsProceedings of the 12th Working Conference on Mining Software Repositories10.5555/2820518.2820547(235-245)Online publication date: 16-May-2015
https://dl.acm.org/doi/10.5555/2820518.2820547
Ozkaya INord RKoziolek HAvgeriou P(2015)Toward Simpler, not Simplistic, Quantification of Software Architecture and MetricsACM SIGSOFT Software Engineering Notes10.1145/2815021.281503740:5(43-46)Online publication date: 14-Sep-2015
https://dl.acm.org/doi/10.1145/2815021.2815037
Nord ROzkaya IKoziolek HAvgeriou P(2014)Quantifying software architecture quality report on the first international workshop on software architecture metricsACM SIGSOFT Software Engineering Notes10.1145/2659118.265914039:5(32-34)Online publication date: 17-Sep-2014
https://dl.acm.org/doi/10.1145/2659118.2659140

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents