research-article

Effects of Hints on Debugging Scratch Programs: An Empirical Study with Primary School Teachers in Training

Authors:

Luisa Greifenstein,

Florian Obermueller,

Ewald Wasmeier,

Gordon FraserAuthors Info & Claims

WiPSCE '21: Proceedings of the 16th Workshop in Primary and Secondary Computing Education

Article No.: 3, Pages 1 - 10

https://doi.org/10.1145/3481312.3481344

Published: 19 October 2021 Publication History

Abstract

Bugs in learners’ programs are often the result of fundamental misconceptions. Teachers frequently face the challenge of first having to understand such bugs, and then suggest ways to fix them. In order to enable teachers to do so effectively and efficiently, it is desirable to support them in recognising and fixing bugs. Misconceptions often lead to recurring patterns of similar bugs, enabling automated tools to provide this support in terms of hints on occurrences of common bug patterns. In this paper, we investigate to what extent the hints improve the effectiveness and efficiency of teachers in debugging learners’ programs using a cohort of 163 primary school teachers in training, tasked to correct buggy programs, with and without hints on bug patterns. Our experiment suggests that automatically generated hints can reduce the effort of finding and fixing bugs from 8.66 to 5.24 minutes, while increasing the effectiveness by 34% more correct solutions. While this improvement is convincing, arguably teachers in training might first need to learn debugging “the hard way” to not miss the opportunity to learn by relying on tools. We therefore investigate whether the use of hints during training affects their ability to recognise and fix bugs without hints. Our experiment provides no significant evidence that either learning to debug with hints or learning to debug “the hard way” leads to better learning effects. Overall, this suggests that bug patterns might be a useful concept to include in the curriculum for teachers in training, while tool-support to recognise these patterns is desirable for teachers in practice.

References

[1]

Vincent Aleven, Ido Roll, Bruce M McLaren, and Kenneth R Koedinger. 2016. Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education 26, 1(2016), 205–223.

[2]

Manfred Max Bergman. 2010. Hermeneutic content analysis: Textual and audiovisual analyses within a mixed methods framework. SAGE Handbook of Mixed Methods in Social and Behavioral Research (2nd ed.). Thousand Oaks, CA: SAGE(2010), 379–396.

[3]

Bryce Boe, Charlotte Hill, Michelle Len, Greg Dreschler, Phillip Conrad, and Diana Franklin. 2013. Hairball: Lint-inspired static analysis of scratch projects. SIGCSE 2013 - Proceedings of the 44th ACM Technical Symposium on Computer Science Education, 215–220. https://doi.org/10.1145/2445196.2445265

Digital Library

[4]

G. Fraser, U. Heuer, N. Körber, F. Obermüller, and E. Wasmeier (accepted). 2021. LitterBox: A Linter for Scratch Programs. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE).

[5]

Christoph Frädrich, Florian Obermüller, Nina Körber, Ute Heuer, and Gordon Fraser. 2020. Common Bugs in Scratch Programs. In ITiCSE ’20: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education. 89–95. https://doi.org/10.1145/3341525.3387389

Digital Library

[6]

Luke Gusukuma, Austin Cory Bart, Dennis Kafura, and Jeremy Ernst. 2018. Misconception-Driven Feedback: Results from an Experimental Study. In Proceedings of the 2018 ACM Conference on International Computing Education Research (Espoo, Finland) (ICER ’18). Association for Computing Machinery, New York, NY, USA, 160–168. https://doi.org/10.1145/3230977.3231002

Digital Library

[7]

Maurice H Halstead. 1977. Elements of software science. (1977).

[8]

Stuart Hansen and Erica Eddy. 2007. Engagement and Frustration in Programming Projects. In Proceedings of the 38th SIGCSE Technical Symposium on Computer Science Education(SIGCSE ’07). ACM, New York, NY, USA, 271–275. https://doi.org/10.1145/1227310.1227407

Digital Library

[9]

Felienne Hermans and Efthimia Aivaloglou. 2016. Do code smells hamper novice programming? A controlled experiment on Scratch programs. In 2016 IEEE 24th International Conference on Program Comprehension (ICPC). 1–10. https://doi.org/10.1109/ICPC.2016.7503706

[10]

Manu Kapur and Katerine Bielaczyc. 2012. Designing for productive failure. Journal of the Learning Sciences 21, 1 (2012), 45–83.

[11]

Cazembe Kennedy, Aubrey Lawson, Yvon Feaster, and Eileen Kraemer. 2020. Misconception-Based Peer Feedback: A Pedagogical Technique for Reducing Misconceptions. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (Trondheim, Norway) (ITiCSE ’20). Association for Computing Machinery, New York, NY, USA, 166–172. https://doi.org/10.1145/3341525.3387392

Digital Library

[12]

ChanMin Kim, Jiangmei Yuan, Lucas Vasconcelos, Minyoung Shin, and Roger B Hill. 2018. Debugging during block-based programming. Instructional Science 46, 5 (2018), 767–787.

[13]

Irene Lee, Fred Martin, Jill Denner, Bob Coulter, Walter Allan, Jeri Erickson, Joyce Malyn-Smith, and Linda Werner. 2011. Computational thinking for youth in practice. Acm Inroads 2, 1 (2011), 32–37.

Digital Library

[14]

John Maloney, Mitchel Resnick, Natalie Rusk, Brian Silverman, and Evelyn Eastmond. 2010. The Scratch Programming Language and Environment. ACM Transactions on Computing Education (TOCE) 10 (11 2010), 16. https://doi.org/10.1145/1868358.1868363

Digital Library

[15]

Samiha Marwan, Anay Dombe, and Thomas W. Price. 2020. Unproductive Help-Seeking in Programming: What It is and How to Address It. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (Trondheim, Norway) (ITiCSE ’20). Association for Computing Machinery, New York, NY, USA, 54–60. https://doi.org/10.1145/3341525.3387394

Digital Library

[16]

Samiha Marwan, Joseph Jay Williams, and Thomas Price. 2019. An Evaluation of the Impact of Automated Programming Hints on Performance and Learning. In Proceedings of the 2019 ACM Conference on International Computing Education Research. 61–70.

Digital Library

[17]

Tilman Michaeli and Ralf Romeike. 2019. Current Status and Perspectives of Debugging in the K12 Classroom: A Qualitative Study. In 2019 IEEE Global Engineering Education Conference (EDUCON). IEEE, 1030–1038.

[18]

Jesús Moreno-León, Gregorio Robles, and Marcos Román-González. 2015. Dr. Scratch: Automatic Analysis of Scratch Projects to Assess and Foster Computational Thinking. RED-Revista de Educación a Distancia (09 2015).

[19]

Benjamin Paaßen, Barbara Hammer, Thomas Price, Tiffany Barnes, Sebastian Gross, and Niels Pinkwart. 2017. The Continuous Hint Factory - Providing Hints in Vast and Sparsely Populated Edit Distance Spaces. (08 2017).

[20]

Thomas Price, Rui Zhi, and Tiffany Barnes. 2017. Evaluation of a Data-driven Feedback Algorithm for Open-ended Programming. In Proceedings of the 10th International Conference on Educational Data Mining. 192–197.

[21]

Thomas W. Price, Yihuan Dong, and Tiffany Barnes. 2016. Generating Data-driven Hints for Open-ended Programming. In EDM. 191–198.

[22]

Thomas W. Price, Zhongxiu Liu, Veronica Cateté, and Tiffany Barnes. 2017. Factors Influencing Students’ Help-Seeking Behavior While Programming with Human and Computer Tutors. In Proceedings of the 2017 ACM Conference on International Computing Education Research (Tacoma, Washington, USA) (ICER ’17). Association for Computing Machinery, New York, NY, USA, 127–135. https://doi.org/10.1145/3105726.3106179

Digital Library

[23]

Kelly Rivers and Kenneth Koedinger. 2015. Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor. International Journal of Artificial Intelligence in Education 27 (10 2015). https://doi.org/10.1007/s40593-015-0070-z

[24]

Sue Sentance and Andrew Csizmadia. 2017. Computing in the curriculum: Challenges and strategies from a teacher’s perspective. Education and Information Technologies 22, 2 (2017), 469–495.

Digital Library

[25]

Rishabh Singh, Sumit Gulwani, and Armando Solar-Lezama. 2012. Automated Feedback Generation for Introductory Programming Assignments. ACM SIGPLAN Notices 48 (04 2012). https://doi.org/10.1145/2491956.2462195

Digital Library

[26]

Juha Sorva. 2018. Misconceptions and the Beginner Programmer. 171–187.

[27]

Andreas Stahlbauer, Marvin Kreis, and Gordon Fraser. 2019. Testing scratch programs automatically. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 165–175.

Digital Library

[28]

Alaaeddin Swidan, Felienne Hermans, and Marileen Smit. 2018. Programming Misconceptions for School Students. In Proceedings of the 2018 ACM Conference on International Computing Education Research (Espoo, Finland) (ICER ’18). Association for Computing Machinery, New York, NY, USA, 151–159. https://doi.org/10.1145/3230977.3230995

Digital Library

[29]

Peeratham Techapalokul and Eli Tilevich. 2017. Quality Hound — An online code smell analyzer for scratch programs. In 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 337–338. https://doi.org/10.1109/VLHCC.2017.8103498

[30]

Wengran Wang, Yudong Rao, Rui Zhi, Samiha Marwan, Ge Gao, and Thomas W. Price. 2020. Step Tutor: Supporting Students through Step-by-Step Example-Based Feedback. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education (Trondheim, Norway) (ITiCSE ’20). Association for Computing Machinery, New York, NY, USA, 391–397. https://doi.org/10.1145/3341525.3387411

Digital Library

[31]

Aman Yadav, Sarah Gretter, Susanne Hambrusch, and Phil Sands. 2016. Expanding computer science education in schools: understanding teacher experiences and challenges. Computer Science Education 26, 4 (2016), 235–254.

[32]

K. Zimmerman and C. R. Rupakheti. 2015. An Automated Framework for Recommending Program Elements to Novices (N). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). 283–288.

Digital Library

Cited By

Yang SBaird MO’Rourke EBrennan KSchneider B(2024)Decoding Debugging Instruction: A Systematic Literature Review of Debugging InterventionsACM Transactions on Computing Education10.1145/369065224:4(1-44)Online publication date: 5-Sep-2024
https://dl.acm.org/doi/10.1145/3690652
Deiner AFraser GRoychoudhury APaiva AAbreu RStorey M(2024)NuzzleBug: Debugging Block-Based Programs in ScratchProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3623331(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3623331
Deiner AFeldmeier PFraser GSchweikl SWang W(2023)Automated test generation for Scratch programsEmpirical Software Engineering10.1007/s10664-022-10255-x28:3Online publication date: 13-May-2023
https://dl.acm.org/doi/10.1007/s10664-022-10255-x

Index Terms

Effects of Hints on Debugging Scratch Programs: An Empirical Study with Primary School Teachers in Training
1. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Index terms have been assigned to the content through auto-classification.

Recommendations

Code Perfumes: Reporting Good Code to Encourage Learners
WiPSCE '21: Proceedings of the 16th Workshop in Primary and Secondary Computing Education

Block-based programming languages like enable children to be creative while learning to program. Even though the block-based approach simplifies the creation of programs, learning to program can nevertheless be challenging. Automated tools such as ...
Common Bugs in Scratch Programs
ITiCSE '20: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education

Bugs in SCRATCH programs can spoil the fun and inhibit learning success. Many common bugs are the result of recurring patterns of bad code. In this paper we present a collection of common code patterns that typically hint at bugs in SCRATCH programs, ...
NuzzleBug: Debugging Block-Based Programs in Scratch
ICSE '24: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering

While professional integrated programming environments support developers with advanced debugging functionality, block-based programming environments for young learners often provide no support for debugging at all, thus inhibiting debugging and ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

WiPSCE '21: Proceedings of the 16th Workshop in Primary and Secondary Computing Education

October 2021

119 pages

ISBN:9781450385718

DOI:10.1145/3481312

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WiPSCE '21

WiPSCE '21: The 16th Workshop in Primary and Secondary Computing Education

October 18 - 20, 2021

Virtual Event, Germany

Acceptance Rates

Overall Acceptance Rate 104 of 279 submissions, 37%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
233
Total Downloads

Downloads (Last 12 months)81
Downloads (Last 6 weeks)13

Reflects downloads up to 21 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yang SBaird MO’Rourke EBrennan KSchneider B(2024)Decoding Debugging Instruction: A Systematic Literature Review of Debugging InterventionsACM Transactions on Computing Education10.1145/369065224:4(1-44)Online publication date: 5-Sep-2024
https://dl.acm.org/doi/10.1145/3690652
Deiner AFraser GRoychoudhury APaiva AAbreu RStorey M(2024)NuzzleBug: Debugging Block-Based Programs in ScratchProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3623331(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3623331
Deiner AFeldmeier PFraser GSchweikl SWang W(2023)Automated test generation for Scratch programsEmpirical Software Engineering10.1007/s10664-022-10255-x28:3Online publication date: 13-May-2023
https://dl.acm.org/doi/10.1007/s10664-022-10255-x

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents