Investigating developers’ perception on software testability and its effects
Authors: 
Tushar Sharma, 
Stefanos Georgiou, Maria Kechagia, Taher A. Ghaleb, 
Federica SarroAuthors Info & Claims
Abstract
The opinions and perspectives of software developers are highly regarded in software engineering research. The experience and knowledge of software practitioners are frequently sought to validate assumptions and evaluate software engineering tools, techniques, and methods. However, experimental evidence may unveil further or different insights, and in some cases even contradict developers’ perspectives. In this work, we investigate the correlation between software developers’ perspectives and experimental evidence about testability smells (i.e., programming practices that may reduce the testability of a software system). Specifically, we first elicit opinions and perspectives of software developers through a questionnaire survey on a catalog of four testability smells, we curated for this work. We also extend our tool DesigniteJava to automatically detect these smells in order to gather empirical evidence on testability smells. To this end we conduct a large-scale empirical study on Java repositories containing approximately 46 million lines of code to investigate the relationship of testability smells with test quality, number of tests, and reported bugs. Our results show that testability smells do not correlate with test smells at the class granularity or with test suit size. Furthermore, we do not find a causal relationship between testability smells and bugs. Moreover, our results highlight that the empirical evidence does not match developers’ perspective on testability smells. Thus, suggesting that despite developers’ invaluable experience, their opinions and perspectives might need to be complemented with empirical evidence before bringing it into practice. This further confirms the importance of data-driven software engineering, which advocates the need and value of ensuring that all design and development decisions are supported by data.
References
[1]
Al-Subaihin AA, Sarro F, Black S, Capra L, and Harman M App store effects on software engineering practices IEEE Trans Softw Eng 2021 47 2 300-319
[2]
Alenezi M, Zarour M (2018) An empirical study of bad smells during software evolution using designite tool. i-Manager’s Journal on Software Engineering 12(4): 12
[3]
Aljedaani W, Peruma A, Aljohani A, Alotaibi M, Mkaouer MW, Ouni A, Newman CD, Ghallab A, Ludi S (2021) Test smell detection tools: A systematic mapping study. In Evaluation and Assessment in Software Engineering, EASE 2021, New York, NY, USA, Association for Computing Machinery page 170–180
[4]
Bavota G, Qusef A, Oliveto R, De Lucia A, Binkley D (2012) An empirical analysis of the distribution of unit test smells and their impact on software maintenance. In 2012 28th IEEE International Conference on Software Maintenance (ICSM), pages 56–65
[5]
Bavota G, Qusef A, Oliveto R, Lucia A, and Binkley D Are test smells really harmful? an empirical study Empirical Softw. Engg 2015 20 4 1052-1094
[6]
Berry KJ, Paul J, Mielke W (1988) A Generalization of Cohen’s Kappa Agreement Measure to Interval Measurement and Multiple Raters. Educational and Psychological Measurement 48(4): 921–933 eprint:
[7]
Binder RV Design for testability in object-oriented systems Commun. ACM 1994 37 9 87-101
[8]
Bruntink M and van Deursen A An empirical study into class testability J Syst Softw 2006 79 9 1219-1232
[9]
Chowdhary V (2009) Practicing testability in the real world. In 2009 International Conference on Software Testing Verification and Validation, pages 260–268
[10]
Couto C, Pires P, Valente MT, da Silva Bigonha R, Hora AC, Anquetil N (2013) Bugmaps-granger: A tool for causality analysis between source code metrics and bugs
[11]
Cox D, Miller H (1965) The Theory of Stochastic Process. Chapman and Hall, London, 1 edition
[12]
Deursen AV, Moonen L, Bergh A, Kok G (2001) Refactoring test code. In Proceedings of the 2nd International Conference on Extreme Programming and Flexible Processes in Software Engineering (XP2001, pages 92–95
[13]
Devanbu P, Zimmermann T, Bird C (2016) Belief & evidence in empirical software engineering. In Proceedings of the 38th International Conference on Software Engineering, ICSE ’16, New York, NY, USA, Association for Computing Machinery page 108–119
[14]
Eck M, Palomba F, Castelluccio M, Bacchelli A (2019) Understanding flaky tests: The developer’s perspective. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE, New York, NY, USA, Association for Computing Machinery page 830–840
[15]
Eposhi A, Oizumi W, Garcia A, Sousa L, Oliveira R, Oliveira A (2019) Removal of design problems through refactorings: Are we looking at the right symptoms? In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC), pages 148–153
[16]
Fatima S, Ghaleb TA, Briand L (2022) Flakify: A black-box, language model-based predictor for flaky tests. IEEE Trans Softw Eng
[17]
Feathers M (2004) Working Effectively with Legacy Code: WORK EFFECT LEG CODE_p1. Prentice Hall Professional
[18]
Filho FGS, Lelli V, Santos IdS, Andrade RMC (2020) Correlations among software testability metrics. In 19th Brazilian Symposium on Software Quality, SBQS’20, New York, NY, USA, Association for Computing Machinery
[19]
Freedman R Testability of software components IEEE Transactions on Software Engineering 1991 17 6 553-564
[20]
Fuller WA (1976) Introduction to Statistical Time Series. John Wiley and Sons New York, 1 edition
[21]
Garousi V, Felderer M, and Kılıçaslan FN A survey on software testability Information and Software Technology 2019 108 35-64
[22]
Garousi V, Felderer M, Mäntylä MV (2018) Guidelines for including grey literature and conducting multivocal literature reviews in software engineering
[23]
Garousi V and Küçük B Smells in software test code: A survey of knowledge in industry and academia Journal of Systems and Software 2018 138 52-81
[24]
Granger CWJ Investigating causal relations by econometric models and cross-spectral methods Econometrica 1969 37 3 424-438
[25]
Hassan MM, Afzal W, Blom M, Lindström B, Andler SF, Eldh S (2015) Testability and sofware robustness: A systematic literature review. In 2015 41st Euromicro Conference on Software Engineering and Advanced Applications, pages 341–348
[26]
Hevery M (2008) Writing Testable Code. https://testing.googleblog.com/2008/08/by-miko-hevery-so-you-decided-to.html
[27]
Human M (2022) Why You Should Be Replacing Full Stack Tests with Ember Tests. https://www.mutuallyhuman.com/blog/why-you-should-be-replacing-full-stack-tests-with-ember-tests/
[28]
Janes AA, Succi G (2012) The dark side of agile software development. In Proceedings of the ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, Onward! 2012, New York, NY, USA. Association for Computing Machinery page 215–228
[29]
Jeffrey VM (1991) Factors that affect software testability. Technical report
[30]
Junior NS, Rocha L, Martins LA, Machado I (2020) A survey on test practitioners’ awareness of test smells
[31]
Kaczanowski T (2013) Practical Unit Testing with JUnit and Mockito. Tomasz Kaczanowski
[32]
Khan RA and Mustafa K Metric based testability model for object oriented design (mtmood) SIGSOFT Softw. Eng. Notes 2009 34 2 1-6
[33]
Kim DJ, Chen T-HP, and Yang J The secret life of test smells - an empirical study on test smell evolution and maintenance Empirical Software Engineering 2021 26 5 100
[34]
Kolb R, Muthig D (2006) Making testing product lines more efficient by improving the testability of product line architectures. In Proceedings of the ISSTA 2006 Workshop on Role of Software Architecture for Testing and Analysis, ROSATEA ’06, New York, NY, USA. Association for Computing Machinery pages 22–27
[35]
Kwiatkowski D, Phillips PC, Schmidt P, and Shin Y Testing the null hypothesis of stationarity against the alternative of a unit root: How sure are we that economic time series have a unit root? Journal of Econometrics 1992 54 1 159-178
[36]
Le Traon Y, Robach C (1995) From hardware to software testability. In Proceedings of 1995 IEEE International Test Conference (ITC), pages 710–719
[37]
Le Traon Y, Robach C (1997) Testability measurements for data flow designs. In Proceedings Fourth International Software Metrics Symposium, pages 91–98
[38]
Lienberherr KJ Formulations and benefits of the law of demeter SIGPLAN Not 1989 24 3 67-78
[39]
Lo B, Shi H (1998) A preliminary testability model for object-oriented software. In Proceedings. 1998 International Conference Software Engineering: Education and Practice (Cat. No.98EX220), pages 330–337
[40]
Marshall L, Webber J (2000) Gotos considered harmful and other programmers’ taboos. Department of Computing Science Technical Report Series
[41]
Mouchawrab S and Briand LC Labiche YA (2005) measurement framework for object-oriented software testability Information and Software Technology, Most Cited Journal Articles in Software Engineering - 1999 47 15 979-997
[42]
Munaiah N, Kroh S, Cabrey C, and Nagappan M Curating GitHub for engineered software projects Empirical Software Engineering 2017 22 6 3219-3253
[43]
Murphy-Hill E, Parnin C, and Black AP How we refactor, and how we know it IEEE Transactions on Software Engineering 2012 38 1 5-18
[44]
Nguyen T, Delaunay M, Robach C (2002) Testability analysis for software components. In International Conference on Software Maintenance, Proceedings pages 422–429
[45]
Nguyen TB, Delaunay M, and Robach C Testability Analysis of Data-Flow Software Electronic Notes in Theoretical Computer Science 2005 116 213-225
[46]
Oizumi W, Sousa L, Oliveira A, Carvalho L, Garcia A, Colanzi T, Oliveira R (2019) On the density and diversity of degradation symptoms in refactored classes: A multi-case study. In 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE), IEEE pages 346–357
[47]
Oliveira P, Lima FP, Valente MT, Serebrenik A (2014) Rttool: A tool for extracting relative thresholds for source code metrics. In 2014 IEEE International Conference on Software Maintenance and Evolution, pages 629–632
[48]
Oliveira P, Valente MT, Lima FP (2014) Extracting relative thresholds for source code metrics. In 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering (CSMR-WCRE), pages 254–263
[49]
Palomba F, Bavota G, Penta MD, Fasano F, Oliveto R, and Lucia AD A large-scale empirical study on the lifecycle of code smell co-occurrences Information and Software Technology 2018 99 1-10
[50]
Payne JE, Alexander RT, and Hutchinson CD Design-for-testability for object-oriented software. Object Magazine 1997 7 5 34-43
[51]
Peruma A, Almalki K, Newman CD, Mkaouer MW, Ouni A, Palomba F (2020) Tsdetect: An open source test smells detection tool. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, page 1650–1654, New York, NY, USA, Association for ComputingMachinery
[52]
Pettichord B (2002) Design for testability. In Pacific Northwest Software Quality Conference, pages 1–28
[53]
Pina D, Seaman C, Goldman A (2022) Technical debt prioritization: A developer’s perspective. In Proceedings of the International Conference on Technical Debt, TechDebt ’22, Association for Computing Machinery, New York, NY, USA page 46–55
[54]
Rahman F, Bird C, Devanbu P (2010) Clones: What is that smell? In 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010), pages 72–81
[55]
Ribeiro DM, da Silva FQB, Valença D, Freitas ELSX, França C (2016) Advantages and disadvantages of using shared code from the developers perspective: A qualitative study. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM ’16, New York, NY, USA, Association for Computing Machinery
[56]
Sharma T (2018) DesigniteJava. https://github.com/tushartushar/DesigniteJava
[57]
Sharma T, Georgiou S, Kechagia M, Ghaleb TA, Sarro F (2022) Replication Package for Testability Study. https://github.com/SMART-Dal/testability
[58]
Sharma T, Singh P, and Spinellis D An empirical investigation on the relationship between design and architecture smells Empirical Software Engineering 2020 25 5 4020-4068
[59]
Singh PK, Sangwan OP, Singh AP, and Pratap A An assessment of software testability using fuzzy logic technique for aspect-oriented software International Journal of Information Technology and Computer Science (IJITCS) 2015 7 3 18
[60]
Spadini D, Palomba F, Zaidman A, Bruntink M, Bacchelli A (2018) On the relation of test smells to software code quality. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME) pages 1–12
[61]
Spearman C (1961) The proof and measurement of association between two things
[62]
Suryanarayana G, Samarthyam G, Sharma T (2014) Refactoring for Software Design Smells: Managing Technical Debt. Morgan Kaufmann, 1 edition
[63]
Sward RE, Chamillard A (2004) Re-engineering global variables in ada. In Proceedings of the 2004 annual ACM SIGAda international conference on Ada: The engineering of correct and reliable software for real-time & distributed systems using Ada and related technologies, pages 29–34
[64]
Terragni V, Salza P, Pezzé M (2020) Measuring software testability modulo test quality. In Proceedings of the 28th International Conference on Program Comprehension, ICPC ’20 page 241–251
[65]
Thomas D and Hunt A The Pragmatic Programmer: your journey to mastery 2019 Addison-Wesley Professional
[66]
Toner B The impact of agreement bias on the ranking of questionnaire response J Soc Psychol 1987 127 2 221-222
[67]
Tufano M, Palomba F, Bavota G, Di Penta M, Oliveto R, De Lucia A, Poshyvanyk D (2016) An empirical investigation into the nature of test smells. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, ASE ’16, New York, NY, USA, Association for Computing Machinery pages 4–15
[68]
Uchôa A, Barbosa C, Oizumi W, Blenilio P, Lima R, Garcia A, Bezerra C (2020) How does modern code review impact software design degradation? an in-depth empirical study. In 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 511–522
[69]
Vincent J, King G, Lay P, and Kinghorn J Principles of Built-In-Test for Run-Time-Testability in Component-Based Software Systems Softw Qual J 2002 10 2 115-133
[70]
Virgínio T, Martins L, Rocha L, Santana R, Cruz A, Costa H, Machado I (2020) Jnose: Java test smell detector. In Proceedings of the 34th Brazilian Symposium on Software Engineering, SBES ’20, New York, NY, USA, Association for Computing Machinery page 564–569
[71]
Voas JM (1996) Object-Oriented Software Testability, Springer US, Boston, MA pages 279–290
[72]
Vranken H, Witteman M, and Van Wuijtswinkel R Design for testability in hardware software systems IEEE Design Test of Computers 1996 13 3 79-86
[73]
Zhao L (2006) A new approach for software testability analysis. In Proceedings of the 28th International Conference on Software Engineering, ICSE ’06, page 985–988
[74]
Zhou Y, Leung H, Song Q, Zhao J, Lu H, Chen L, and Xu B An in-depth investigation into the relationships between structural metrics and unit testability in object-oriented systems Science china information sciences 2012 55 12 2800-2815
[75]
Zilberfeld G (2012) Design for Testability – The True Story. https://www.infoq.com/articles/Testability/
[76]
Faruk Arar Ömer and Ayan K Deriving thresholds of software metrics to predict faults on open source software: Replicated case studies Expert Systems with Applications 2021 61 106-121
Recommendations
An empirical study on the effects of code visibility on program testability
Software testability represents the degree of ease with which a software artifact supports testing. When it is easy to detect defects in a program through testing, the program has high testability; otherwise, the testability of the program is low. As an ...
An Empirical Study on Effects of Code Visibility on Code Coverage of Software Testing
AST '15: Proceedings of the 2015 IEEE/ACM 10th International Workshop on Automation of Software TestSoftware testability is the degree of difficulty to test a program. Code visibility is important to support design principles, such as information hiding. It is widely believed that code visibility has effects on testability. However, little empirical ...
An empirical study on effects of code visibility on code coverage of software testing
AST '15: Proceedings of the 10th International Workshop on Automation of Software TestSoftware testability is the degree of difficulty to test a program. Code visibility is important to support design principles, such as information hiding. It is widely believed that code visibility has effects on testability. However, little empirical ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Kluwer Academic Publishers
United States
Publication History
Published: 13 September 2023
Accepted: 27 July 2023
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 27 Feb 2025