Evaluating the Effectiveness of a Testing Checklist Intervention in CS2: An Quasi-experimental Replication Study
Pages 55 - 64
Abstract
Students often run into trouble when learning and practicing software testing. Recent prior studies demonstrate that a lightweight testing checklist that contains testing strategies and tutorial information could assist students in writing higher-quality tests. Prior studies also suggest that students with lower prior knowledge in unit testing may benefit more from the checklists. However, insights on the potential benefits and costs of the testing checklists in a classroom setting are lacking. To address this, we conducted an operational replication study in a CS2 course with 342 students (171 from Fall 2023 and 171 from Spring 2024) who had no prior experience in unit testing.
In this paper, we report our experience in introducing the testing checklists as optional tool support in a CS2 course. To evaluate the effectiveness of the testing checklists in a classroom setting, we quantitatively analyze a combination of programming assignment submissions and survey responses generated by students. Our results suggest that students who received the testing checklists achieved significantly higher quality in their test code, in terms of code coverage and mutation coverage, compared to those who did not. We also observed that the exposure to the testing checklists in students’ early learning process encouraged students to write more unit tests to cover possible testing scenarios.
References
[1]
[n. d.]. EclEmma: Coverage Counters. https://www.eclemma.org/jacoco/trunk/doc/counters.html. Accessed: 2024-06-06.
[2]
[n. d.]. PITest: Mutation Operators. http://pitest.org/quickstart/mutators/. Accessed: 2024-06-06.
[3]
ACM, 2013. Computer Science Curricula Recommendations: Curriculum Guidelines for Undergraduate Degree Programs in Computer Science. https://www.acm.org/education/curricula-recommendations.
[4]
Alireza Ahadi, Arto Hellas, Petri Ihantola, Ari Korhonen, and Andrew Petersen. 2016. Replication in computing education research: researcher attitudes and experiences. In Proceedings of the 16th Koli Calling International Conference on Computing Education Research(Koli Calling 2016). ACM.
[5]
Tiago L. Alves and Joost Visser. 2009. Static Estimation of Test Coverage. In International Working Conference on Source Code Analysis and Manipulation. 55–64.
[6]
Michael Andersson. 2017. An Experimental Evaluation of PIT’s Mutation Operators., 27 pages.
[7]
Maurício Aniche, Felienne Hermans, and Arie van Deursen. 2019. Pragmatic Software Testing Education. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (Minneapolis, MN, USA) (SIGCSE ’19). ACM, New York, NY, USA, 414–420.
[8]
D. Athanasiou, A. Nugroho, J. Visser, and A. Zaidman. 2014. Test Code Quality and Its Relation to Issue Handling Performance. IEEE Transactions on Software Engineering 40, 11 (Nov 2014), 1100–1125.
[9]
Gina R. Bai, Kai Presler-Marshall, Thomas Price, and Kathryn T. Stolee. 2022. Check It Off: Exploring the Impact of a Checklist Intervention on the Quality of Student-written Unit Tests. In 27th ACM Conference on Innovation and Technology in Computer Science Education(ITiCSE ’22).
[10]
Gina R. Bai, Justin Smith, and Kathryn T. Stolee. 2021. How Students Unit Test: Perceptions, Practices, and Pitfalls. In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1 (Virtual Event, Germany) (ITiCSE ’21). ACM, New York, NY, USA, 248–254.
[11]
Gina R. Bai, Sandeep Sthapit, Sarah Heckman, Thomas W. Price, and Kathryn T. Stolee. 2023. An Experience Report on Introducing Explicit Strategies into Testing Checklists for Advanced Beginners. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (, Turku, Finland,) (ITiCSE 2023). ACM, New York, NY, USA, 194–200.
[12]
Lex Bijlsma, Niels Doorn, Harrie Passier, Harold Pootjes, and Sylvia Stuurman. 2021. How do Students Test Software Units?. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET). 189–198.
[13]
Raquel Blanco, Manuel Trinidad, María José Suárez-Cabal, Alejandro Calderón, Mercedes Ruiz, and Javier Tuya. 2023. Can gamification help in software testing education? Findings from an empirical study. Journal of Systems and Software 200 (2023), 111647.
[14]
Michael K. Bradshaw. 2015. Ante Up: A Framework to Strengthen Student-Based Testing of Assignments. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (Kansas City, Missouri, USA) (SIGCSE ’15). ACM, New York, NY, USA, 488–493.
[15]
David I. Broadhurst and Douglas B. Kell. 2007. Statistical strategies for avoiding false discoveries in metabolomics and related experiments. Metabolomics 2 (2007), 171–196. https://api.semanticscholar.org/CorpusID:7638542
[16]
Neil C. C. Brown, Eva Marinus, and Aleata Hubbard Cheuoua. 2022. Launching Registered Report Replications in Computer Science Education Research. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1(ICER 2022). ACM.
[17]
Ingrid A. Buckley and Winston S. Buckley. 2017. Teaching Software Testing using Data Structures. International Journal of Advanced Computer Science and Applications 8, 4 (2017).
[18]
Kevin Buffardi and Juan Aguirre-Ayala. 2021. Unit Test Smells and Accuracy of Software Engineering Student Test Suites. ACM, New York, NY, USA, 234–240.
[19]
Michael E Caspersen and Jens Bennedsen. 2007. Instructional design of a programming course: a learning theoretic approach. In Proceedings of the third international workshop on Computing education research. 111–122.
[20]
Seth Chaiklin. 2003. The Zone of Proximal Development in Vygotsky’s Analysis of Learning and Instruction. Vygotsky’s Educational Theory in Cultural Context (09 2003).
[21]
Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016. PIT: A Practical Mutation Testing Tool for Java (Demo). In Proceedings of the 25th International Symposium on Software Testing and Analysis (Saarbrücken, Germany) (ISSTA 2016). Association for Computing Machinery, New York, NY, USA, 449–452.
[22]
Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4 (1994), 253–278.
[23]
Ermira Daka and Gordon Fraser. 2014. A Survey on Unit Testing Practices and Problems. In 2014 IEEE International Symposium on Software Reliability Engineering (ISSRE ’14). 201–211.
[24]
Simone C. dos Santos, Maria da Conceição Moraes Batista, Ana Paula C. Cavalcanti, Jones O. Albuquerque, and Silvio R.L. Meira. 2009. Applying PBL in Software Engineering Education. In 2009 22nd Conference on Software Engineering Education and Training. 182–189.
[25]
Stephen H. Edwards, Zalia Shams, Michael Cogswell, and Robert C. Senkbeil. 2012. Running Students’ Software Tests against Each Others’ Code: New Life for an Old "Gimmick". In Proceedings of the 43rd ACM Technical Symposium on Computer Science Education (Raleigh, North Carolina, USA) (SIGCSE ’12). ACM, New York, NY, USA, 221–226.
[26]
Gordon Fraser, Alessio Gambi, Marvin Kreis, and José Miguel Rojas. 2019. Gamifying a Software Testing Course with Code Defenders. In Proc. of the ACM Technical Symposium on Computer Science Education (SIGCSE)(SIGCSE’19). ACM.
[27]
Alessio Gaspar, Sarah Langevin, Naomi Boyer, and Ralph Tindell. 2013. A Preliminary Review of Undergraduate Programming Students’ Perspectives on Writing Tests, Working with Others, & Using Peer Testing. In ACM SIGITE Conference on Information Technology Education. 109–114.
[28]
Michael H. Goldwasser. 2002. A Gimmick to Integrate Software Testing Throughout the Curriculum. In Technical Symposium on Computer Science Education(SIGCSE ’02). 271–275.
[29]
Giovanni Grano, Fabio Palomba, and Harald C. Gall. 2019. Lightweight Assessment of Test-Case Effectiveness using Source-Code-Quality Indicators. IEEE Transactions on Software Engineering (2019), 1–1.
[30]
Omar S. Gómez, Natalia Juristo, and Sira Vegas. 2014. Understanding replication of experiments in software engineering: A classification. Information and Software Technology 56, 8 (2014), 1033–1048.
[31]
Qiang Hao, David H. Smith IV, Naitra Iriumi, Michail Tsikerdekis, and Amy J. Ko. 2019. A Systematic Investigation of Replications in Computing Education Research. ACM Transactions on Computing Education 19, 4 (Aug. 2019), 1–18.
[32]
Spencer E. Harpe. 2015. How to analyze Likert and other rating scale data. Currents in Pharmacy Teaching and Learning 7, 6 (2015), 836–850.
[33]
Sarah Heckman, Jessica Young Schmidt, and Jason King. 2020. Integrating Testing Throughout the CS Curriculum. In Conference on Software Testing, Verification and Validation Workshops (ICSTW). 441–444.
[34]
Edward L. Jones. 2001. Integrating Testing into the Curriculum — Arsenic in Small Doses. In Proceedings of the 32nd SIGCSE Technical Symposium on Computer Science Education(SIGCSE ’01). ACM, New York, NY, USA, 337–341.
[35]
Liesbeth Kester, Fred Paas, and Jeroen J. G. Van Merrienboer. 2010. Instructional Control of Cognitive Load in the Design of Complex Learning Environments. Cognitive Load Theory (04 2010).
[36]
Ken Koster. 2008. A State Coverage Tool for JUnit. In Companion of the 30th International Conference on Software Engineering(ICSE Companion ’08). 965–966.
[37]
Otávio Augusto Lazzarini Lemos, Fábio Fagundes Silveira, Fabiano Cutigi Ferrari, and Alessandro Garcia. 2018. The impact of Software Testing education on code reliability: An empirical assessment. Journal of Systems and Software 137 (2018), 497–511.
[38]
Lauren E Margulieux and Richard Catrambone. 2016. Improving problem solving with subgoal labels in expository text and worked examples. Learning and Instruction 42 (2016), 58–71.
[39]
Lauren E Margulieux, Briana B Morrison, and Adrienne Decker. 2020. Reducing withdrawal and failure rates in introductory programming with subgoal labeled worked examples. International Journal of STEM Education 7 (2020), 1–16.
[40]
Lauren E Margulieux, Briana B Morrison, Baker Franke, and Harivololona Ramilison. 2020. Effect of Implementing Subgoals in Code. org’s Intro to Programming Unit in Computer Science Principles. ACM Transactions on Computing Education (TOCE) 20, 4 (2020), 1–24.
[41]
Samiha Marwan, Bita Akram, Tiffany Barnes, and Thomas W Price. 2022. Adaptive immediate feedback for block-based programming: Design and evaluation. IEEE Transactions on Learning Technologies 15, 3 (2022), 406–420.
[42]
Monica M. McGill. 2019. Discovering Empirically-Based Best Practices in Computing Education Through Replication, Reproducibility, and Meta-Analysis Studies. In Proceedings of the 19th Koli Calling International Conference on Computing Education Research(Koli Calling ’19). ACM.
[43]
Briana B Morrison, Lauren E Margulieux, Barbara Ericson, and Mark Guzdial. 2016. Subgoals help students solve Parsons problems. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education. 42–47.
[44]
Fabio Palomba, Annibale Panichella, Andy Zaidman, Rocco Oliveto, and Andrea De Lucia. 2016. Automatic Test Case Generation: What if Test Code Quality Matters?. In International Symposium on Software Testing and Analysis(ISSTA 2016). 130–141.
[45]
Kai Petersen and Jefferson Seide Molléri. 2021. Preliminary Evaluation of a Survey Checklist in the Context of Evidence-based Software Engineering Education. In Proceedings of the 16th International Conference on Evaluation of Novel Approaches to Software Engineering, ENASE 2021, Online Streaming, April 26-27, 2021, Raian Ali, Hermann Kaindl, and Leszek A. Maciaszek (Eds.). SCITEPRESS, 437–444.
[46]
Raphael Pham, Stephan Kiesling, Olga Liskin, Leif Singer, and Kurt Schneider. 2014. Enablers, Inhibitors, and Perceptions of Testing in Novice Software Teams. In ACM SIGSOFT International Symposium on Foundations of Software Engineering(FSE 2014). 30–40.
[47]
Leandro Sales Pinto, Saurabh Sinha, and Alessandro Orso. 2012. Understanding Myths and Realities of Test-suite Evolution. In Foundations of Software Engineering(FSE ’12). ACM, Article 33, 11 pages.
[48]
Upsorn Praphamontripong, Mark Floryan, and Ryan Ritzo. 2020. A Preliminary Report on Hands-On and Cross-Course Activities in a College Software Testing Course. In Conference on Software Testing, Verification and Validation Workshops (ICSTW). 445–451.
[49]
Guoping Rong, Jingyi Li, Mingjuan Xie, and Tao Zheng. 2012. The Effect of Checklist in Code Review for Inexperienced Students: An Empirical Study. In 2012 IEEE 25th Conference on Software Engineering Education and Training. 120–124.
[50]
Guoping Rong, Jingyi Li, Mingjuan Xie, and Tao Zheng. 2012. The effect of checklist in code review for inexperienced students: An empirical study. In 2012 IEEE 25th Conference on Software Engineering Education and Training. IEEE, 120–124.
[51]
Lilian Passos Scatalon, Jeffrey C. Carver, Rogério Eduardo Garcia, and Ellen Francine Barbosa. 2019. Software Testing in Introductory Programming Courses: A Systematic Mapping Study. In ACM Technical Symposium on Computer Science Education(SIGCSE ’19). 421–427.
[52]
Lilian Passos Scatalon, Jorge Marques Prates, Draylson Micael de Souza, Ellen Francine Barbosa, and Rogério Eduardo Garcia. 2017. Towards the Role of Test Design in Programming Assignments. In 2017 IEEE 30th Conference on Software Engineering Education and Training (CSEE&T). 170–179.
[53]
Matthew Sibbald, Anique BH De Bruin, and Jeroen JG van Merrienboer. 2014. Finding and fixing mistakes: do checklists work for clinicians with different levels of experience?Advances in Health Sciences Education 19 (2014), 43–51.
[54]
Matthew Sibbald, Anique B H de Bruin, and Jeroen J G van Merrienboer. 2013. Checklists improve experts’ diagnostic decisions. Medical Education 47, 3 (2013), 301–308.
[55]
Jaime Spacco and William Pugh. 2006. Helping Students Appreciate Test-driven Development (TDD). In Companion to the 21st ACM SIGPLAN Symposium on Object-oriented Programming Systems, Languages, and Applications (Portland, Oregon, USA) (OOPSLA ’06). ACM, New York, NY, USA, 907–913.
[56]
John Sweller. 2011. CHAPTER TWO - Cognitive Load Theory. Psychology of Learning and Motivation, Vol. 55. Academic Press, 37–76.
[57]
Janneke van de Pol, Monique Volman, and Jos Beishuizen. 2010. Scaffolding in Teacher–Student Interaction: A Decade of Research. Educational Psychology Review 22 (09 2010), 271–296.
[58]
Michael Wick, Daniel Stevenson, and Paul Wagner. 2005. Using Testing and JUnit across the Curriculum. In Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education (St. Louis, Missouri, USA) (SIGCSE ’05). ACM, New York, NY, USA, 236–240.
[59]
David Wood, Jerome S Bruner, and Gail Ross. 1976. The role of tutoring in problem solving. Journal of child psychology and psychiatry 17, 2 (1976), 89–100.
[60]
Rui Zhi, Thomas W Price, Samiha Marwan, Alexandra Milliken, Tiffany Barnes, and Min Chi. 2019. Exploring the impact of worked examples in a novice programming environment. In Proceedings of the 50th acm technical symposium on computer science education. 98–104.
Recommendations
Check It Off: Exploring the Impact of a Checklist Intervention on the Quality of Student-authored Unit Tests
ITiCSE '22: Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1Software testing is an essential skill for computer science students. Prior work reports that students desire support in determining what code to test and which scenarios should be tested. In response to this, we present a lightweight testing checklist ...
An Experience Report on Introducing Explicit Strategies into Testing Checklists for Advanced Beginners
ITiCSE 2023: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1Software testing is a critical skill for computing students, but learning and practicing testing can be challenging, particularly for beginners. A recent study suggests that a lightweight testing checklist that contains testing strategies and tutorial ...
How Students Unit Test: Perceptions, Practices, and Pitfalls
ITiCSE '21: Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1Unit testing is reported as one of the skills that graduating students lack, yet it is an essential skill for professional software developers. Understanding the challenges students face during testing can help inform practices for software testing ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
August 2024
539 pages
ISBN:9798400704758
DOI:10.1145/3632620
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 12 August 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- NSF IUSE
Conference
ICER 2024
Sponsor:
ICER 2024: ACM Conference on International Computing Education Research
August 13 - 15, 2024
VIC, Melbourne, Australia
Acceptance Rates
Overall Acceptance Rate 189 of 803 submissions, 24%
Upcoming Conference
ICER 2025
- Sponsor:
- sigcse
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 157Total Downloads
- Downloads (Last 12 months)157
- Downloads (Last 6 weeks)44
Reflects downloads up to 13 Feb 2025
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in