research-article

An experimental evaluation of test driven development vs. test-last development with industry professionals

Authors:

Krzysztof Wnuk,

Misagh MoayyedAuthors Info & Claims

EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

Article No.: 50, Pages 1 - 10

https://doi.org/10.1145/2601248.2601267

Published: 13 May 2014 Publication History

Abstract

Test-Driven Development (TDD) is a software development approach where test cases are written before actual development of the code in iterative cycles. Context: TDD has gained attention of many software practitioners during the last decade since it has contributed several benefits to the software development process. However, empirical evidence of its dominance in terms of internal code quality, external code quality and productivity is fairly limited. Objective: The aim behind conducting this controlled experiment with professional Java developers is to see the impact of Test-Driven Development (TDD) on internal code quality, external code quality and productivity compared to Test-Last Development (TLD). Results: Experiment results indicate that values found related to number of acceptance test cases passed, McCabe's Cyclomatic complexity, branch coverage, number of lines of code per person hours, number of user stories implemented per person hours are statistically insignificant. However, static code analysis results were found statistically significant in the favor of TDD. Moreover, the results of the survey revealed that the majority of developers in the experiment prefer TLD over TDD, given the lesser required level of learning curve as well as the minimum effort needed to understand and employ TLD compared to TDD.

References

[1]

C. Larman and V. R. Basili. Iterative and incremental developments. a brief history. Computer, 36(6):47--56, June 2003.

Digital Library

[2]

Kent Beck and Cynthia Andres. Extreme Programming Explained: Embrace Change. Addison-Wesley Professional, 2 edition, November 2004.

Digital Library

[3]

M. Pancur, M. Ciglaric, M. Trampus, and T. Vidmar. Towards empirical evaluation of test-driven development in a university environment, volume 2. IEEE, 2003.

[4]

H. Cibulski and A. Yehudai. Regression test selection techniques for test-driven development. In 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pages 115--124. IEEE, March 2011.

Digital Library

[5]

Liang Huang and Mike Holcombe. Empirical investigation towards the effectiveness of test first programming. Information and Software Technology, 51(1):182--194, 2009.

Digital Library

[6]

S. Ambler. Quality in an agile world. Software Quality Professional, 7(4):34, 2005.

[7]

Sumanth Yenduri and Louise A Perkins. Impact of using test-driven development: A case study. In Software Engineering Research and Practice, pages 126--129, 2006.

[8]

Thirumalesh Bhat and Nachiappan Nagappan. Evaluating the efficacy of test-driven development: Industrial case studies. In ISCE'06 - 5th ACM-IEEE International Symposium on Empirical Software Engineering, volume 2006 of ISESE'06 - Proceedings of the 5th ACM-IEEE International Symposium on Empirical Software Engineering, pages 356--363, Rio de Janeiro, Brazil, 2006. ACM.

Digital Library

[9]

M. M. Muller and O. Hagner. Experiment about test-first programming. IEE Proceedings-Software, 149(5):131--6, October 2002.

[10]

Atul Gupta and Pankaj Jalote. An experimental evaluation of the effectiveness and efficiency of the test driven development. In 1st International Symposium on Empirical Software Engineering and Measurement, ESEM, Proceedings - 1st International Symposium on Empirical Software Engineering and Measurement, ESEM 2007, pages 285--294, Madrid, Spain, 2007. Inst. of Elec. and Elec. Eng. Computer Society.

Digital Library

[11]

Matjaz Pancur and Mojca Ciglaric. Impact of test-driven development on productivity, code and tests: A controlled experiment. Information and Software Technology, 53(6):557--573, 2011.

Digital Library

[12]

Sami Kollanus. Test-driven development - still a promising approach? In 7th International Conference on the Quality of Information and Communications Technology, QUATIC, Proceedings - 7th International Conference on the Quality of Information and Communications Technology, QUATIC 2010, pages 403--408, Porto, Portugal, 2010. IEEE Computer Society.

Digital Library

[13]

Hakan Erdogmus, Maurizio Morisio, and Marco Torchiano. On the effectiveness of the test-first approach to programming. IEEE Transactions on Software Engineering, 31(3):226--237, 2005.

Digital Library

[14]

A. Geras, M. Smith, and J. Miller. A prototype empirical evaluation of test driven development. In Proceedings - 10th International Symposium on Software Metrics, METRICS 2004, September 14, 2004 - September 16, 2004, Proceedings - International Software Metrics Symposium, pages 405--416, Chicago, IL, United states, 2004. IEEE Computer Society.

Digital Library

[15]

Boby George and Laurie Williams. A structured experiment of test-driven development. volume 46 of Information and Software Technology, pages 337--342. Elsevier, 2004.

[16]

E. M. Maximilien and L. Williams. Assessing test-driven development at IBM. In IEEE 25th International Conference on Software Engineering, 3-10 May 2003, pages 564--9, Los Alamitos, CA, USA, 2003. IEEE Comput. Soc.

Digital Library

[17]

Hussan Munir, Misagh Moayyed, and Kai Petersen. Considering rigor and relevance when evaluating test driven development: A systematic review. Information and Software Technology, (0):--, 2014.

Digital Library

[18]

Forrest Shull, Grigori Melnik, Burak Turhan, Lucas Layman, Madeline Diep, and Hakan Erdogmus. What do we know about test-driven development? IEEE Software, 27(6):16--19, 2010.

Digital Library

[19]

M. Pancur, M. Ciglaric, M. Trampus, and T. Vidmar. Towards empirical evaluation of test-driven development in a university environment. In Proc. IEEE Region 8 EUROCON 2003 - Computer as a Tool, volume vol.2 of IEEE Region 8 EUROCON 2003. Computer as a Tool. Proceedings (Cat. No.03EX655), pages 83--6, Piscataway, NJ, USA, 2003. IEEE.

[20]

Boby George and Laurie Williams. An initial investigation of test driven development in industry. In Proceedings of the 2003 ACM Symposium on Applied Computing, March 9, 2003 - March 12, 2003, Proceedings of the ACM Symposium on Applied Computing, pages 1135--1139, Melbourne, FL, United states, 2003. Association for Computing Machinery.

Digital Library

[21]

Gerardo Canfora, Aniello Cimitile, Felix Garcia, Mario Piattini, and Corrado Aaron Visaggio. Evaluating advantages of test driven development: A controlled experiment with professionals. In ISCE'06 - 5th ACM-IEEE International Symposium on Empirical Software Engineering, September 21, 2006 - September 22, 2006, volume 2006, pages 364--371, Rio de Janeiro, Brazil, 2006. Association for Computing Machinery.

Digital Library

[22]

Lars-Ola Damm and Lars Lundberg. Results from introducing component-level test automation and test-driven development. Journal of Systems and Software, 79(7):1001--1014, 2006.

Digital Library

[23]

Nachiappan Nagappan, E. Michael Maximilien, Thirumalesh Bhat, and Laurie Williams. Realizing quality improvement through test driven development: Results and experiences of four industrial teams. Empirical Software Engineering, 13(3):289--302, 2008.

Digital Library

[24]

L. Williams, E. M. Maximilien, and M. Vouk. Test-driven development as a defect-reduction practice. In 14th International Symposium on Software Reliability Engineering, 17-20 Nov. 2003, 14th International Symposium on Software Reliability Engineering, pages 34--45, Los Alamitos, CA, USA, 2003. IEEE Comput. Soc.

Digital Library

[25]

Claes Wohlin, Per Runeson, and Martin Host. Experimentation in Software Engineering: An Introduction. Springer, 1st edition, December 1999.

Digital Library

[26]

Eclipse IDE for java developers | eclipse packages.

[27]

Downloads for KentBeck's junit - GitHub.

[28]

Robert C. Martin. Agile Software Development, Principles, Patterns, and Practices. Prentice Hall, 1st edition, October 2002.

Digital Library

[29]

Norman E. Fenton and Shari Lawrence Pfleeger. Software Metrics: A Rigorous and Practical Approach, Revised. Course Technology, 2 edition, February 1998.

Digital Library

[30]

Brian Henderson-Sellers. Object-Oriented Metrics: Measures of Complexity. Prentice Hall, 1 edition, December 1995.

Digital Library

[31]

John Wilder Tukey. Exploratory Data Analysis. Addison-Wesley Publishing Company, 1977.

[32]

Sam Kash Kachigan. Statistical Analysis: An Interdisciplinary Introduction to Univariate & Multivariate Methods. Radius Press, January 1986.

[33]

T. Dyba, V. B. Kampenes, and D. I. K. Sjoberg. A systematic review of statistical power in software engineering experiments. Information and Software Technology, 48(8):745--55, 2006.

[34]

C. Desai, D. Janzen, and K. Savage. A survey of evidence for test-driven development in academia. SIGCSE Bulletin, 40(2):97--101, June 2008.

Digital Library

[35]

David S. Janzen and Hossein Saiedian. A leveled examination of test-driven development acceptance. In 29th International Conference on Software Engineering, ICSE 2007, May 20, 2007 - May 26, 2007, Proceedings - International Conference on Software Engineering, pages 719--722, Minneapolis, MN, United states, 2007. Inst. of Elec. and Elec. Eng. Computer Society.

Digital Library

[36]

D. S. Janzen and H. Saiedian. Does test-driven development really improve software design quality? Software, IEEE, 25(2):77--84, 2008.

Digital Library

[37]

David S. Janzen and Hossein Saiedian. On the influence of test-driven development on software design. volume 2006, pages 141--148, Turtle Bay, HI, United states, 2006.

Digital Library

[38]

Lech Madeyski. The impact of test-first programming on branch coverage and mutation score indicator of unit tests: An experiment. Information and Software Technology, 52(2):169--184, 2010.

Digital Library

[39]

L. Madeyski and L. Szala. The impact of test-driven development on software development productivity an empirical study. Software Process Improvement, pages 200--211, 2007.

Digital Library

[40]

John Huan Vu, Niklas Frojd, Clay Shenkel-Therolf, and David S. Janzen. In 6th International Conference on Information Technology: New Generations, ITNG, pages 229--234, Las Vegas, NV, United states, 2009. IEEE Computer Society.

[41]

Thomas Flohr and Thorsten Schneider. Lessons learned from an XP experiment with students: Test-first needs more teachings. In Product-Focused Software Process Improvement, volume 4034, pages 305--318. Springer Berlin Heidelberg, Berlin, Heidelberg, 2006.

Digital Library

Cited By

Chen ZLi GCao YDu C(2024)Research on the Application of Agile Testing in Satellite Ground Control SystemProceedings of the International Conference on Machine Learning, Pattern Recognition and Automation Engineering10.1145/3696687.3696688(1-8)Online publication date: 7-Aug-2024
https://dl.acm.org/doi/10.1145/3696687.3696688
Uyaguari FAcuña SCastro JFucci DDieste OVegas S(2024)Relevant Information in TDD Experiment ReportingACM Transactions on Software Engineering and Methodology10.1145/368883734:2(1-41)Online publication date: 22-Aug-2024
https://dl.acm.org/doi/10.1145/3688837
Zhu TLiu ZXu TTang ZZhang TPan MXia XBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)Exploring and Improving Code Completion for Test CodeProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644421(137-148)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644421
Show More Cited By

Index Terms

An experimental evaluation of test driven development vs. test-last development with industry professionals

Recommendations

The effectiveness of test-driven development: an industrial case study

Test-driven development (TDD) is a software development practice, where test cases are incrementally written before implementing the production code. This paper presents the results of a multi-case study investigating the effectiveness of TDD within an ...
Test-Driven Development: Concepts, Taxonomy, and Future Direction

Test-driven development creates software in very short iterations with minimal upfront design. Poised for widespread adoption TDD has become the focus of an increasing number of researchers and developers.
Test-driven development with mutation testing – an experimental study
Abstract
Test-driven development (TDD) is a popular design approach used by the developers with testing being the important software development driving factor. On the other hand, mutation testing is considered one of the most effective testing techniques. ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

May 2014

486 pages

ISBN:9781450324762

DOI:10.1145/2601248

General Chair:
Martin Shepperd
Brunel University, UK
,
Program Chairs:
Tracy Hall
Brunel University
,
Ingunn Myrtveit
The Norwegian Business School

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Brunel University: Brunel University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Swedish National Science Foundation

Conference

EASE '14

Sponsor:

Brunel University

EASE '14: 18th International Conference on Evaluation and Assessment in Software Engineering

May 13 - 14, 2014

England, London, United Kingdom

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
846
Total Downloads

Downloads (Last 12 months)43
Downloads (Last 6 weeks)3

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen ZLi GCao YDu C(2024)Research on the Application of Agile Testing in Satellite Ground Control SystemProceedings of the International Conference on Machine Learning, Pattern Recognition and Automation Engineering10.1145/3696687.3696688(1-8)Online publication date: 7-Aug-2024
https://dl.acm.org/doi/10.1145/3696687.3696688
Uyaguari FAcuña SCastro JFucci DDieste OVegas S(2024)Relevant Information in TDD Experiment ReportingACM Transactions on Software Engineering and Methodology10.1145/368883734:2(1-41)Online publication date: 22-Aug-2024
https://dl.acm.org/doi/10.1145/3688837
Zhu TLiu ZXu TTang ZZhang TPan MXia XBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)Exploring and Improving Code Completion for Test CodeProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644421(137-148)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644421
Vihovde EMeng Q(2024)Test-Driven Development: Ensuring Code Quality in Integration with CI/CD2024 8th International Conference on Management Engineering, Software Engineering and Service Sciences (ICMSS)10.1109/ICMSS61211.2024.00009(8-11)Online publication date: 12-Jan-2024
https://doi.org/10.1109/ICMSS61211.2024.00009
Priya SKahsai TGurfinkel A(2024)Unlocking the Power of Environment Assumptions for Unit ProofsSoftware Engineering and Formal Methods10.1007/978-3-031-77382-2_21(366-384)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1007/978-3-031-77382-2_21
Ren WBarrett S(2023)Test‐driven development, engagement in activity, and maintainabilityIET Software10.1049/sfw2.1213517:4(509-525)Online publication date: 27-Jul-2023
https://dl.acm.org/doi/10.1049/sfw2.12135
Papis BGrochowski KSubzda KSijko K(2022)Experimental Evaluation of Test-Driven Development With Interns Working on a Real Industrial ProjectIEEE Transactions on Software Engineering10.1109/TSE.2020.302752248:5(1644-1664)Online publication date: 1-May-2022
https://doi.org/10.1109/TSE.2020.3027522
Staegemann DVolk MPohl MHaertel CHintsch JTurowski K(2022)Identifying Guidelines for Test-Driven Development in Software Engineering—A Literature ReviewProceedings of Seventh International Congress on Information and Communication Technology10.1007/978-981-19-2397-5_30(327-336)Online publication date: 17-Aug-2022
https://doi.org/10.1007/978-981-19-2397-5_30
Karac ITurhan BJuristo N(2021)A Controlled Experiment with Novice Developers on the Impact of Task Description Granularity on Software Quality in Test-Driven DevelopmentIEEE Transactions on Software Engineering10.1109/TSE.2019.292037747:7(1315-1330)Online publication date: 1-Jul-2021
https://doi.org/10.1109/TSE.2019.2920377
Abushama HAlassam HElhaj F(2021)The effect of Test-Driven Development and Behavior-Driven Development on Project Success Factors: A Systematic Literature Review Based Study2020 International Conference on Computer, Control, Electrical, and Electronics Engineering (ICCCEEE)10.1109/ICCCEEE49695.2021.9429593(1-9)Online publication date: 26-Feb-2021
https://doi.org/10.1109/ICCCEEE49695.2021.9429593
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten