research-article

An experimental evaluation of test driven development vs. test-last development with industry professionals

Authors:

Hussan Munir,

Krzysztof Wnuk,

Kai Petersen,

Misagh MoayyedAuthors Info & Claims

EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

Article No.: 50, Pages 1 - 10

https://doi.org/10.1145/2601248.2601267

Published: 13 May 2014 Publication History

Get Access

Abstract

Test-Driven Development (TDD) is a software development approach where test cases are written before actual development of the code in iterative cycles. Context: TDD has gained attention of many software practitioners during the last decade since it has contributed several benefits to the software development process. However, empirical evidence of its dominance in terms of internal code quality, external code quality and productivity is fairly limited. Objective: The aim behind conducting this controlled experiment with professional Java developers is to see the impact of Test-Driven Development (TDD) on internal code quality, external code quality and productivity compared to Test-Last Development (TLD). Results: Experiment results indicate that values found related to number of acceptance test cases passed, McCabe's Cyclomatic complexity, branch coverage, number of lines of code per person hours, number of user stories implemented per person hours are statistically insignificant. However, static code analysis results were found statistically significant in the favor of TDD. Moreover, the results of the survey revealed that the majority of developers in the experiment prefer TLD over TDD, given the lesser required level of learning curve as well as the minimum effort needed to understand and employ TLD compared to TDD.

References

[1]

C. Larman and V. R. Basili. Iterative and incremental developments. a brief history. Computer, 36(6):47--56, June 2003.

Digital Library

Google Scholar

[2]

Kent Beck and Cynthia Andres. Extreme Programming Explained: Embrace Change. Addison-Wesley Professional, 2 edition, November 2004.

Digital Library

Google Scholar

[3]

M. Pancur, M. Ciglaric, M. Trampus, and T. Vidmar. Towards empirical evaluation of test-driven development in a university environment, volume 2. IEEE, 2003.

Google Scholar

[4]

H. Cibulski and A. Yehudai. Regression test selection techniques for test-driven development. In 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pages 115--124. IEEE, March 2011.

Digital Library

Google Scholar

[5]

Liang Huang and Mike Holcombe. Empirical investigation towards the effectiveness of test first programming. Information and Software Technology, 51(1):182--194, 2009.

Digital Library

Google Scholar

[6]

S. Ambler. Quality in an agile world. Software Quality Professional, 7(4):34, 2005.

Google Scholar

[7]

Sumanth Yenduri and Louise A Perkins. Impact of using test-driven development: A case study. In Software Engineering Research and Practice, pages 126--129, 2006.

Google Scholar

[8]

Thirumalesh Bhat and Nachiappan Nagappan. Evaluating the efficacy of test-driven development: Industrial case studies. In ISCE'06 - 5th ACM-IEEE International Symposium on Empirical Software Engineering, volume 2006 of ISESE'06 - Proceedings of the 5th ACM-IEEE International Symposium on Empirical Software Engineering, pages 356--363, Rio de Janeiro, Brazil, 2006. ACM.

Digital Library

Google Scholar

[9]

M. M. Muller and O. Hagner. Experiment about test-first programming. IEE Proceedings-Software, 149(5):131--6, October 2002.

Crossref

Google Scholar

[10]

Atul Gupta and Pankaj Jalote. An experimental evaluation of the effectiveness and efficiency of the test driven development. In 1st International Symposium on Empirical Software Engineering and Measurement, ESEM, Proceedings - 1st International Symposium on Empirical Software Engineering and Measurement, ESEM 2007, pages 285--294, Madrid, Spain, 2007. Inst. of Elec. and Elec. Eng. Computer Society.

Digital Library

Google Scholar

[11]

Matjaz Pancur and Mojca Ciglaric. Impact of test-driven development on productivity, code and tests: A controlled experiment. Information and Software Technology, 53(6):557--573, 2011.

Digital Library

Google Scholar

[12]

Sami Kollanus. Test-driven development - still a promising approach? In 7th International Conference on the Quality of Information and Communications Technology, QUATIC, Proceedings - 7th International Conference on the Quality of Information and Communications Technology, QUATIC 2010, pages 403--408, Porto, Portugal, 2010. IEEE Computer Society.

Digital Library

Google Scholar

[13]

Hakan Erdogmus, Maurizio Morisio, and Marco Torchiano. On the effectiveness of the test-first approach to programming. IEEE Transactions on Software Engineering, 31(3):226--237, 2005.

Digital Library

Google Scholar

[14]

A. Geras, M. Smith, and J. Miller. A prototype empirical evaluation of test driven development. In Proceedings - 10th International Symposium on Software Metrics, METRICS 2004, September 14, 2004 - September 16, 2004, Proceedings - International Software Metrics Symposium, pages 405--416, Chicago, IL, United states, 2004. IEEE Computer Society.

Digital Library

Google Scholar

[15]

Boby George and Laurie Williams. A structured experiment of test-driven development. volume 46 of Information and Software Technology, pages 337--342. Elsevier, 2004.

Google Scholar

[16]

E. M. Maximilien and L. Williams. Assessing test-driven development at IBM. In IEEE 25th International Conference on Software Engineering, 3-10 May 2003, pages 564--9, Los Alamitos, CA, USA, 2003. IEEE Comput. Soc.

Digital Library

Google Scholar

[17]

Hussan Munir, Misagh Moayyed, and Kai Petersen. Considering rigor and relevance when evaluating test driven development: A systematic review. Information and Software Technology, (0):--, 2014.

Digital Library

Google Scholar

[18]

Forrest Shull, Grigori Melnik, Burak Turhan, Lucas Layman, Madeline Diep, and Hakan Erdogmus. What do we know about test-driven development? IEEE Software, 27(6):16--19, 2010.

Digital Library

Google Scholar

[19]

M. Pancur, M. Ciglaric, M. Trampus, and T. Vidmar. Towards empirical evaluation of test-driven development in a university environment. In Proc. IEEE Region 8 EUROCON 2003 - Computer as a Tool, volume vol.2 of IEEE Region 8 EUROCON 2003. Computer as a Tool. Proceedings (Cat. No.03EX655), pages 83--6, Piscataway, NJ, USA, 2003. IEEE.

Crossref

Google Scholar

[20]

Boby George and Laurie Williams. An initial investigation of test driven development in industry. In Proceedings of the 2003 ACM Symposium on Applied Computing, March 9, 2003 - March 12, 2003, Proceedings of the ACM Symposium on Applied Computing, pages 1135--1139, Melbourne, FL, United states, 2003. Association for Computing Machinery.

Digital Library

Google Scholar

[21]

Gerardo Canfora, Aniello Cimitile, Felix Garcia, Mario Piattini, and Corrado Aaron Visaggio. Evaluating advantages of test driven development: A controlled experiment with professionals. In ISCE'06 - 5th ACM-IEEE International Symposium on Empirical Software Engineering, September 21, 2006 - September 22, 2006, volume 2006, pages 364--371, Rio de Janeiro, Brazil, 2006. Association for Computing Machinery.

Digital Library

Google Scholar

[22]

Lars-Ola Damm and Lars Lundberg. Results from introducing component-level test automation and test-driven development. Journal of Systems and Software, 79(7):1001--1014, 2006.

Digital Library

Google Scholar

[23]

Nachiappan Nagappan, E. Michael Maximilien, Thirumalesh Bhat, and Laurie Williams. Realizing quality improvement through test driven development: Results and experiences of four industrial teams. Empirical Software Engineering, 13(3):289--302, 2008.

Digital Library

Google Scholar

[24]

L. Williams, E. M. Maximilien, and M. Vouk. Test-driven development as a defect-reduction practice. In 14th International Symposium on Software Reliability Engineering, 17-20 Nov. 2003, 14th International Symposium on Software Reliability Engineering, pages 34--45, Los Alamitos, CA, USA, 2003. IEEE Comput. Soc.

Digital Library

Google Scholar

[25]

Claes Wohlin, Per Runeson, and Martin Host. Experimentation in Software Engineering: An Introduction. Springer, 1st edition, December 1999.

Digital Library

Google Scholar

[26]

Eclipse IDE for java developers | eclipse packages.

Google Scholar

[27]

Downloads for KentBeck's junit - GitHub.

Google Scholar

[28]

Robert C. Martin. Agile Software Development, Principles, Patterns, and Practices. Prentice Hall, 1st edition, October 2002.

Digital Library

Google Scholar

[29]

Norman E. Fenton and Shari Lawrence Pfleeger. Software Metrics: A Rigorous and Practical Approach, Revised. Course Technology, 2 edition, February 1998.

Digital Library

Google Scholar

[30]

Brian Henderson-Sellers. Object-Oriented Metrics: Measures of Complexity. Prentice Hall, 1 edition, December 1995.

Digital Library

Google Scholar

[31]

John Wilder Tukey. Exploratory Data Analysis. Addison-Wesley Publishing Company, 1977.

Google Scholar

[32]

Sam Kash Kachigan. Statistical Analysis: An Interdisciplinary Introduction to Univariate & Multivariate Methods. Radius Press, January 1986.

Google Scholar

[33]

T. Dyba, V. B. Kampenes, and D. I. K. Sjoberg. A systematic review of statistical power in software engineering experiments. Information and Software Technology, 48(8):745--55, 2006.

Crossref

Google Scholar

[34]

C. Desai, D. Janzen, and K. Savage. A survey of evidence for test-driven development in academia. SIGCSE Bulletin, 40(2):97--101, June 2008.

Digital Library

Google Scholar

[35]

David S. Janzen and Hossein Saiedian. A leveled examination of test-driven development acceptance. In 29th International Conference on Software Engineering, ICSE 2007, May 20, 2007 - May 26, 2007, Proceedings - International Conference on Software Engineering, pages 719--722, Minneapolis, MN, United states, 2007. Inst. of Elec. and Elec. Eng. Computer Society.

Digital Library

Google Scholar

[36]

D. S. Janzen and H. Saiedian. Does test-driven development really improve software design quality? Software, IEEE, 25(2):77--84, 2008.

Digital Library

Google Scholar

[37]

David S. Janzen and Hossein Saiedian. On the influence of test-driven development on software design. volume 2006, pages 141--148, Turtle Bay, HI, United states, 2006.

Digital Library

Google Scholar

[38]

Lech Madeyski. The impact of test-first programming on branch coverage and mutation score indicator of unit tests: An experiment. Information and Software Technology, 52(2):169--184, 2010.

Digital Library

Google Scholar

[39]

L. Madeyski and L. Szala. The impact of test-driven development on software development productivity an empirical study. Software Process Improvement, pages 200--211, 2007.

Digital Library

Google Scholar

[40]

John Huan Vu, Niklas Frojd, Clay Shenkel-Therolf, and David S. Janzen. In 6th International Conference on Information Technology: New Generations, ITNG, pages 229--234, Las Vegas, NV, United states, 2009. IEEE Computer Society.

Google Scholar

[41]

Thomas Flohr and Thorsten Schneider. Lessons learned from an XP experiment with students: Test-first needs more teachings. In Product-Focused Software Process Improvement, volume 4034, pages 305--318. Springer Berlin Heidelberg, Berlin, Heidelberg, 2006.

Digital Library

Google Scholar

Cited By

View all

Chen ZLi GCao YDu C(2024)Research on the Application of Agile Testing in Satellite Ground Control SystemProceedings of the International Conference on Machine Learning, Pattern Recognition and Automation Engineering10.1145/3696687.3696688(1-8)Online publication date: 7-Aug-2024
https://dl.acm.org/doi/10.1145/3696687.3696688
Zhu TLiu ZXu TTang ZZhang TPan MXia XBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)Exploring and Improving Code Completion for Test CodeProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644421(137-148)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644421
Vihovde EMeng Q(2024)Test-Driven Development: Ensuring Code Quality in Integration with CI/CD2024 8th International Conference on Management Engineering, Software Engineering and Service Sciences (ICMSS)10.1109/ICMSS61211.2024.00009(8-11)Online publication date: 12-Jan-2024
https://doi.org/10.1109/ICMSS61211.2024.00009
Show More Cited By

Index Terms

An experimental evaluation of test driven development vs. test-last development with industry professionals

Recommendations

The effectiveness of test-driven development: an industrial case study

Test-driven development (TDD) is a software development practice, where test cases are incrementally written before implementing the production code. This paper presents the results of a multi-case study investigating the effectiveness of TDD within an ...
Test-Driven Development: Concepts, Taxonomy, and Future Direction

Test-driven development creates software in very short iterations with minimal upfront design. Poised for widespread adoption TDD has become the focus of an increasing number of researchers and developers.
Test-driven development with mutation testing – an experimental study
Abstract
Test-driven development (TDD) is a popular design approach used by the developers with testing being the important software development driving factor. On the other hand, mutation testing is considered one of the most effective testing techniques. ...

Reviews

Reviewer: Andrew Brooks

The results of an experiment comparing test-driven development (TDD) with test-last development (TLD) are reported. The subjects were professional Java developers who were asked to implement seven small user stories adapted from Robert Martin's bowling game problem. The only statistically significant result obtained was that a static code analysis using the PMD tool came out in favor of TDD. Otherwise, no difference was found between TDD and TLD regarding code quality and developer productivity. The participant dropout rate was very high and usable data was obtained from only 13 out of 31 developers. This meant the statistical analysis was undertaken using groups of seven (TDD) and six (TLD). There is no real surprise that the experiment was found to lack statistical power and that the findings were inconclusive. Also, multiple statistical tests were performed, but no Bonferroni correction was applied. Had such a correction been applied, the result for the static code analysis would no longer be significant. The box plots concerning productivity reveal differences between developers by a factor of three or more. Much might have been learned had the investigators switched from an inferential analysis to an inductive analysis. For example, could the differences in productivity be attributed to varying levels of developer experience and expertise__?__ A major threat to the validity of the experiment, recognized by the investigators themselves, is the fact that the developers performed the task at their places of work at times that suited them. This paper is recommended only to those with an interest in TDD. Online Computing Reviews Service

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

May 2014

486 pages

ISBN:9781450324762

DOI:10.1145/2601248

General Chair:
Martin Shepperd
Brunel University, UK
,
Program Chairs:
Tracy Hall
Brunel University
,
Ingunn Myrtveit
The Norwegian Business School

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Swedish National Science Foundation

Conference

EASE '14

Sponsor:

Brunel University

EASE '14: 18th International Conference on Evaluation and Assessment in Software Engineering

May 13 - 14, 2014

England, London, United Kingdom

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
828
Total Downloads

Downloads (Last 12 months)43
Downloads (Last 6 weeks)4

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Chen ZLi GCao YDu C(2024)Research on the Application of Agile Testing in Satellite Ground Control SystemProceedings of the International Conference on Machine Learning, Pattern Recognition and Automation Engineering10.1145/3696687.3696688(1-8)Online publication date: 7-Aug-2024
https://dl.acm.org/doi/10.1145/3696687.3696688
Zhu TLiu ZXu TTang ZZhang TPan MXia XBaysal OLinares-Vasquez MMoran KSteinmacher I(2024)Exploring and Improving Code Completion for Test CodeProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension10.1145/3643916.3644421(137-148)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643916.3644421
Vihovde EMeng Q(2024)Test-Driven Development: Ensuring Code Quality in Integration with CI/CD2024 8th International Conference on Management Engineering, Software Engineering and Service Sciences (ICMSS)10.1109/ICMSS61211.2024.00009(8-11)Online publication date: 12-Jan-2024
https://doi.org/10.1109/ICMSS61211.2024.00009
Ren WBarrett S(2023)Test‐driven development, engagement in activity, and maintainabilityIET Software10.1049/sfw2.1213517:4(509-525)Online publication date: 27-Jul-2023
https://dl.acm.org/doi/10.1049/sfw2.12135
Papis BGrochowski KSubzda KSijko K(2022)Experimental Evaluation of Test-Driven Development With Interns Working on a Real Industrial ProjectIEEE Transactions on Software Engineering10.1109/TSE.2020.302752248:5(1644-1664)Online publication date: 1-May-2022
https://doi.org/10.1109/TSE.2020.3027522
Staegemann DVolk MPohl MHaertel CHintsch JTurowski K(2022)Identifying Guidelines for Test-Driven Development in Software Engineering—A Literature ReviewProceedings of Seventh International Congress on Information and Communication Technology10.1007/978-981-19-2397-5_30(327-336)Online publication date: 17-Aug-2022
https://doi.org/10.1007/978-981-19-2397-5_30
Karac ITurhan BJuristo N(2021)A Controlled Experiment with Novice Developers on the Impact of Task Description Granularity on Software Quality in Test-Driven DevelopmentIEEE Transactions on Software Engineering10.1109/TSE.2019.292037747:7(1315-1330)Online publication date: 1-Jul-2021
https://doi.org/10.1109/TSE.2019.2920377
Abushama HAlassam HElhaj F(2021)The effect of Test-Driven Development and Behavior-Driven Development on Project Success Factors: A Systematic Literature Review Based Study2020 International Conference on Computer, Control, Electrical, and Electronics Engineering (ICCCEEE)10.1109/ICCCEEE49695.2021.9429593(1-9)Online publication date: 26-Feb-2021
https://doi.org/10.1109/ICCCEEE49695.2021.9429593
Moe MOo K(2020)Consequences of Dependent and Independent Variables based on Acceptance Test Suite Metric Using Test Driven Development Approach2020 IEEE Conference on Computer Applications(ICCA)10.1109/ICCA49400.2020.9022828(1-6)Online publication date: Feb-2020
https://doi.org/10.1109/ICCA49400.2020.9022828
Santos AVegas SUyaguari FDieste OTurhan BJuristo N(2020)Increasing validity through replication: an illustrative TDD caseSoftware Quality Journal10.1007/s11219-020-09512-3Online publication date: 26-Mar-2020
https://doi.org/10.1007/s11219-020-09512-3
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

The effectiveness of test-driven development: an industrial case study

Test-Driven Development: Concepts, Taxonomy, and Future Direction

Test-driven development with mutation testing – an experimental study

Reviews

Access critical reviews of Computing literature here