Distractors Make You Pay Attention: Investigating the Learning Outcomes of Including Distractor Blocks in Parsons Problems
Authors: 
David H. Smith, 
Seth Poulsen, Chinedu Emeka, Zihan Wu, Carl Haynes-Magyar, Craig ZillesAuthors Info & Claims
Pages 177 - 191
Abstract
Background: In CS1 courses, Parsons problems are a popular activity in which students are given blocks of code and asked to rearrange them into the correct order. Parsons problems often include incorrect blocks of code referred to as distractor blocks. Despite their widespread use, there have been few investigations into how distractor blocks impact student learning. Objectives: Our goals are to understand (1) the impact that including distractor blocks in Parsons problems has on learning and (2) the causality underlying that learning, if any. Methods: In this paper, we present the results of an explanatory sequential mixed methods study investigating the impact of distractor blocks on student learning. For the initial, quantitative stage, we use a randomized control trial to quantify the learning outcomes from practice with Parsons problems that include distractor blocks, as measured via post-test taken immediately after the practice activity and a retention test taken a week later. This study is followed by think-aloud interviews with 10 students practicing using a mix of Parsons problems that do and do not contain distractors to understand differences in how students approach those problems. Findings: Our findings show that students who practiced using Parsons problems that contained distractors performed 11 percentage points better on the immediate post-test (statistically significant) and 10 percentage points better on the retention test (approaching significance). The results of the think-aloud interviews indicate that grouping distractors with blocks of correct code causes students to more closely attend to the details of the code within those blocks. Implications: The results of this study indicate that distractors are essential when Parsons problems are used in a formative context. When they are not included, students may be able to successfully place blocks of code without attending to details of the code. This in turn limits their ability to learn new concepts or reinforce existing knowledge from those code blocks.
References
[1]
Robert L Bangert-Drowns, Chen-Lin C Kulik, James A Kulik, and MaryTeresa Morgan. 1991. The instructional effect of feedback in test-like events. Review of educational research 61, 2 (1991), 213–238.
[2]
Brett A Becker and Thomas Fitzpatrick. 2019. What do cs1 syllabi reveal about our expectations of introductory programming students?. In Proceedings of the 50th ACM technical symposium on computer science education. 1011–1017.
[3]
Jeff Bender, Bingpu Zhao, Alex Dziena, and Gail Kaiser. 2023. Integrating Parsons puzzles within Scratch enables efficient computational thinking learning. Research and Practice in Technology Enhanced Learning 18 (2023), 022–022.
[4]
Elizabeth L Bjork, Robert A Bjork, 2011. Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. Psychology and the real world: Essays illustrating fundamental contributions to society 2, 59-68 (2011).
[5]
Elizabeth Ligon Bjork, Nicholas C Soderstrom, and Jeri L Little. 2015. Can multiple-choice testing induce desirable difficulties? Evidence from the laboratory and the classroom. The American Journal of Psychology 128, 2 (2015), 229–239.
[6]
Robert A Bjork. 1994. Memory and metamemory considerations in the training of human beings. (1994).
[7]
Robert A Bjork. 2017. Creating desirable difficulties to enhance learning. Carmarthen: Crown House Publishing (2017).
[8]
Jürgen Börstler, Harald Störrle, Daniel Toll, Jelle Van Assema, Rodrigo Duran, Sara Hooshangi, Johan Jeuring, Hieke Keuning, Carsten Kleiner, and Bonnie MacKellar. 2018. " I know it when I see it" Perceptions of Code Quality: ITiCSE’17 Working Group Report. In Proceedings of the 2017 iticse conference on working group reports. 70–85.
[9]
Derek C Briggs, Alicia C Alonzo, Cheryl Schwab, and Mark Wilson. 2006. Diagnostic assessment with ordered multiple-choice items. Educational Assessment 11, 1 (2006), 33–63.
[10]
James E Bruno and Arie Dirkzwager. 1995. Determining the optimal number of alternatives to a multiple-choice test item: An information theoretic perspective. Educational and Psychological Measurement 55, 6 (1995), 959–966.
[11]
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement 20, 1 (1960), 37–46.
[12]
John W Creswell and Vicki L Plano Clark. 2017. Designing and conducting mixed methods research. Sage publications.
[13]
Lee J Cronbach. 1951. Coefficient alpha and the internal structure of tests. psychometrika 16, 3 (1951), 297–334.
[14]
JHII Cross, T Dean Hendrix, and Larry A Barowski. 2002. Using the debugger as an integral part of teaching CS1. In 32nd Annual Frontiers in Education, Vol. 2. IEEE, F1G–F1G.
[15]
Paul Denny, Andrew Luxton-Reilly, and Beth Simon. 2008. Evaluating a new exam question: Parsons problems. In Proceedings of the fourth international workshop on computing education research. 113–124.
[16]
Yuemeng Du, Andrew Luxton-Reilly, and Paul Denny. 2020. A review of research on Parsons problems. In Proceedings of the Twenty-Second Australasian Computing Education Conference. 195–202.
[17]
Barbara Ericson, Austin McCall, and Kathryn Cunningham. 2019. Investigating the affect and effect of adaptive Parsons problems. In Proceedings of the 19th Koli Calling International Conference on Computing Education Research. 1–10.
[18]
Barbara J Ericson. 2014. Adaptive Parsons problems with discourse rules. In Proceedings of the tenth annual conference on International computing education research. 145–146.
[19]
Barbara J Ericson. 2016. Dynamically adaptive Parsons problems. In Proceedings of the 2016 ACM Conference on International Computing Education Research. 269–270.
[20]
Barbara J Ericson, Paul Denny, James Prather, Rodrigo Duran, Arto Hellas, Juho Leinonen, Craig S Miller, Briana B Morrison, Janice L Pearce, and Susan H Rodger. 2022. Parsons problems and beyond: Systematic literature review and empirical study designs. Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education (2022), 191–234.
[21]
Barbara J Ericson, James D Foley, and Jochen Rick. 2018. Evaluating the efficiency and effectiveness of adaptive Parsons problems. In Proceedings of the 2018 ACM Conference on International Computing Education Research. 60–68.
[22]
Barbara J Ericson, Mark J Guzdial, and Briana B Morrison. 2015. Analysis of interactive features designed to enhance learning in an ebook. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research. 169–178.
[23]
Barbara J Ericson, Lauren E Margulieux, and Jochen Rick. 2017. Solving Parsons problems versus fixing and writing code. In Proceedings of the 17th koli calling international conference on computing education research. 20–29.
[24]
Barbara J Ericson, Janice L Pearce, Susan H Rodger, Andrew Csizmadia, Rita Garcia, Francisco J Gutierrez, Konstantinos Liaskos, Aadarsh Padiyath, Michael James Scott, David H Smith IV, 2023. Multi-Institutional Multi-National Studies of Parsons Problems. In Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education. 57–107.
[25]
Geela Venise Firmalo Fabic, Antonija Mitrovic, and Kourosh Neshatian. 2019. Evaluation of Parsons problems with menu-based self-explanation prompts in a mobile python tutor. International Journal of Artificial Intelligence in Education 29 (2019), 507–535.
[26]
Bridgid Finn and Janet Metcalfe. 2010. Scaffolding feedback to maximize long-term error correction. Memory & cognition 38, 7 (2010), 951–961.
[27]
Max Fowler, David H Smith IV, Mohammed Hassan, Seth Poulsen, Matthew West, and Craig Zilles. 2022. Reevaluating the relationship between explaining, tracing, and writing skills in CS1 in a replication study. Computer Science Education (2022), 1–29.
[28]
Flynn Fromont, Hiruna Jayamanne, and Paul Denny. 2023. Exploring the Difficulty of Faded Parsons Problems for Programming Education. In Proceedings of the 25th Australasian Computing Education Conference. 113–122.
[29]
Mark J Gierl, Okan Bulut, Qi Guo, and Xinxin Zhang. 2017. Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research 87, 6 (2017), 1082–1116.
[30]
Bert F Green, Carolyn R Crone, and Valerie Greaud Folk. 1989. A method for studying differential distractor functioning. Journal of Educational Measurement 26, 2 (1989), 147–160.
[31]
Louis Guttman and Izchak M Schlesinger. 1967. Systematic construction of distractors for ability and achievement test items. Educational and Psychological Measurement 27, 3 (1967), 569–580.
[32]
Qiang Hao, David H Smith IV, Lu Ding, Amy Ko, Camille Ottaway, Jack Wilson, Kai H Arakawa, Alistair Turcan, Timothy Poehlman, and Tyler Greer. 2022. Towards understanding the effective design of automated formative feedback for programming assignments. Computer Science Education 32, 1 (2022), 105–127.
[33]
Qiang Hao, David H Smith IV, Naitra Iriumi, Michail Tsikerdekis, and Amy J Ko. 2019. A systematic investigation of replications in computing education research. ACM Transactions on Computing Education (TOCE) 19, 4 (2019), 1–18.
[34]
Kyle James Harms, Jason Chen, and Caitlin L Kelleher. 2016. Distractors in Parsons problems decrease learning efficiency for young novice programmers. In Proceedings of the 2016 ACM Conference on International Computing Education Research. 241–250.
[35]
Carl C Haynes and Barbara J Ericson. 2021. Problem-Solving Efficiency and Cognitive Load for Adaptive Parsons Problems vs. Writing the Equivalent Code. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.
[36]
Carl Haynes-Magyar. 2024. Neurodiverse Programmers and the Accessibility of Parsons Problems: An Exploratory Multiple-Case Study. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1. 491–497.
[37]
Juha Helminen, Petri Ihantola, Ville Karavirta, and Satu Alaoutinen. 2013. How do students solve Parsons programming problems?–execution-based vs. line-based feedback. In 2013 Learning and Teaching in Computing and Engineering. IEEE, 55–61.
[38]
Xinying Hou, Barbara Jane Ericson, and Xu Wang. 2022. Using adaptive Parsons problems to scaffold write-code problems. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1. 15–26.
[39]
Petri Ihantola and Ville Karavirta. 2011. Two-dimensional parson’s puzzles: The concept, tools, and first observations. Journal of Information Technology Education 10, 2 (2011), 119–132.
[40]
Susan R Jones, Vasti Torres, and Jan Arminio. 2013. Negotiating the complexities of qualitative research in higher education: Fundamental elements and issues. Routledge.
[41]
Sukjin Kang, Lawrence C Scharmann, and Taehee Noh. 2004. Reexamining the role of cognitive conflict in science concept learning. Research in science education 34 (2004), 71–96.
[42]
Manu Kapur. 2008. Productive failure. Cognition and instruction 26, 3 (2008), 379–424.
[43]
Manu Kapur. 2014. Productive failure in learning math. Cognitive science 38, 5 (2014), 1008–1022.
[44]
Manu Kapur and Katerine Bielaczyc. 2012. Designing for productive failure. Journal of the Learning Sciences 21, 1 (2012), 45–83.
[45]
Paul Kirschner. 2018. Inquiry Learning Isn’t–A Call For Direct Explicit Instruction. The researchED Magazine 1, 1 (2018), 9–11.
[46]
Paul Kirschner, John Sweller, and Richard E Clark. 2006. Why unguided learning does not work: An analysis of the failure of discovery learning, problem-based learning, experiential learning and inquiry-based learning. Educational psychologist 41, 2 (2006), 75–86.
[47]
Melina Klepsch and Tina Seufert. 2020. Understanding instructional design effects by differentiated measurement of intrinsic, extraneous, and germane cognitive load. Instructional Science 48, 1 (2020), 45–77.
[48]
Nate Kornell, Matthew Jensen Hays, and Robert A Bjork. 2009. Unsuccessful retrieval attempts enhance subsequent learning.Journal of Experimental Psychology: Learning, Memory, and Cognition 35, 4 (2009), 989.
[49]
Nate Kornell, Patricia Jacobs Klein, and Katherine A Rawson. 2015. Retrieval attempts enhance learning, but retrieval success (versus failure) does not matter.Journal of Experimental Psychology: Learning, Memory, and Cognition 41, 1 (2015), 283.
[50]
James A Kulik and Chen-Lin C Kulik. 1988. Timing of feedback and verbal learning. Review of educational research 58, 1 (1988), 79–97.
[51]
Jeri L Little and Elizabeth Ligon Bjork. 2015. Optimizing multiple-choice tests as tools for learning. Memory & Cognition 43 (2015), 14–26.
[52]
Mike Lopez, Jacqueline Whalley, Phil Robbins, and Raymond Lister. 2008. Relationships between reading, tracing and writing skills in introductory programming. In Proceedings of the fourth international workshop on computing education research. 101–112.
[53]
Anna Ly, John Edwards, Michael Liut, and Andrew Petersen. 2021. Revisiting Syntax Exercises in CS1. In Proceedings of the 22nd Annual Conference on Information Technology Education. 9–14.
[54]
Janet Metcalfe. 2017. Learning from errors. Annual review of psychology 68 (2017), 465–489.
[55]
Brooke C Morin, Krista M Kecskemety, Kathleen A Harper, and Paul Alan Clingan. 2020. Work in Progress: Parsons Problems as a Tool in the First-Year Engineering Classroom. In 2020 ASEE Virtual Annual Conference Content Access.
[56]
Briana B Morrison, Lauren E Margulieux, Barbara Ericson, and Mark Guzdial. 2016. Subgoals help students solve Parsons problems. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education. 42–47.
[57]
Fatni Mufit, Fauzan Festiyed, Ahmad Fauzan, and Lufri Lufri. 2018. Impact of learning model based on cognitive conflict toward student’s conceptual understanding. In IOP Conference Series: Materials Science and Engineering, Vol. 335. IOP Publishing, 012072.
[58]
Linus Östlund, Niklas Wicklund, and Richard Glassey. 2023. It’s Never too Early to Learn About Code Quality: A Longitudinal Study of Code Quality in First-year Computer Science Students. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 792–798.
[59]
Dale Parsons and Patricia Haden. 2006. Parson’s programming puzzles: a fun and effective learning tool for first programming courses. In Proceedings of the 8th Australasian Conference on Computing Education-Volume 52. 157–163.
[60]
Seth Poulsen, Hongxuan Chen, Yael Gertner, Benjamin Cosman, Matthew West, and Geoffrey L Herman. 2023. Measuring the Impact of Distractors on Student Learning Gains while Using Proof Blocks. arXiv preprint arXiv:2311.00792 (2023).
[61]
Seth Poulsen, Yael Gertner, Hongxuan Chen, Benjamin Cosman, Matthew West, and Geoffrey L Herman. 2024. Disentangling the learning gains from reading a book chapter and completing proof blocks problems. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1. 1056–1062.
[62]
Seth Poulsen, Yael Gertner, Benjamin Cosman, Matthew West, and Geoffrey L Herman. 2023. Efficiency of learning from proof blocks versus writing proofs. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 472–478.
[63]
Seth Poulsen, Mahesh Viswanathan, Geoffrey L. Herman, and Matthew West. 2021. Evaluating Proof Blocks Problems as Exam Questions. In Proceedings of the 17th ACM Conference on International Computing Education Research (Virtual Event, USA) (ICER 2021). Association for Computing Machinery, New York, NY, USA, 157–168. https://doi.org/10.1145/3446871.3469741
[64]
Seth Poulsen, Mahesh Viswanathan, Geoffrey L Herman, and Matthew West. 2022. Proof blocks: autogradable scaffolding activities for learning to write proofs. In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1. 428–434.
[65]
Mark R Raymond, Craig Stevens, and S Deniz Bucak. 2019. The optimal number of options for multiple-choice questions on high-stakes tests: application of a revised index for detecting nonfunctional distractors. Advances in Health Sciences Education 24 (2019), 141–150.
[66]
Bob Rehder and Aaron B Hoffman. 2005. Eyetracking and selective attention in category learning. Cognitive psychology 51, 1 (2005), 1–41.
[67]
J Elizabeth Richey, Juan Miguel L Andres-Bray, Michael Mogessie, Richard Scruggs, Juliana MAL Andres, Jon R Star, Ryan S Baker, and Bruce M McLaren. 2019. More confusion and frustration, better learning: The impact of erroneous examples. Computers & Education 139 (2019), 173–190.
[68]
Lindsey E Richland, Nate Kornell, and Liche Sean Kao. 2009. The pretesting effect: Do unsuccessful retrieval attempts enhance learning?Journal of Experimental Psychology: Applied 15, 3 (2009), 243.
[69]
Christopher A Rowland. 2014. The effect of testing versus restudy on retention: a meta-analytic review of the testing effect.Psychological bulletin 140, 6 (2014), 1432.
[70]
Karim Shabani, Mohamad Khatib, and Saman Ebadi. 2010. Vygotsky’s zone of proximal development: Instructional implications and teachers’ professional development.English language teaching 3, 4 (2010), 237–248.
[71]
Anna Shvarts and Arthur Bakker. 2019. The early history of the scaffolding metaphor: Bernstein, Luria, Vygotsky, and before. Mind, Culture, and Activity 26, 1 (2019), 4–23.
[72]
David H Smith, IV, Seth Poulsen, Max Fowler, and Craig Zilles. 2023. Comparing the Impacts of Visually Grouped and Jumbled Distractors on Parsons Problems in CS1 Assessments. In Proceedings of the ACM Conference on Global Computing Education Vol 1. 154–160.
[73]
David H Smith IV. 2023. Useful Distractions? Investigating the Utility of Distractors in Parsons Problems. In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 2. 62–63.
[74]
David H Smith IV, Max Fowler, and Craig Zilles. 2023. Investigating the Role and Impact of Distractors on Parsons Problems in CS1 Assessments. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1. 417–423.
[75]
David H Smith IV and Craig Zilles. 2023. Discovering, autogenerating, and evaluating distractors for python Parsons problems in cs1. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 924–930.
[76]
John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive science 12, 2 (1988), 257–285.
[77]
David Thissen, Lynne Steinberg, and Anne R Fitzpatrick. 1989. Multiple-choice models: The distractors are also part of the item. Journal of Educational Measurement 26, 2 (1989), 161–176.
[78]
Rashmi Vyas and Avinash Supe. 2008. Multiple choice questions: a literature review on the optimal number of options. Natl Med J India 21, 3 (2008), 130–3.
[79]
Lev Semenovich Vygotsky and Michael Cole. 1978. Mind in society: Development of higher psychological processes. Harvard university press.
[80]
Nathaniel Weinman, Armando Fox, and Marti Hearst. 2020. Exploring challenging variations of Parsons problems. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. 1349–1349.
[81]
Nathaniel Weinman, Armando Fox, and Marti A Hearst. 2021. Improving instruction of programming patterns with faded Parsons problems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–4.
[82]
Matthew West, Geoffrey L Herman, and Craig Zilles. 2015. Prairielearn: Mastery-based online problem solving with adaptive scoring and recommendations driven by machine learning. In 2015 ASEE Annual Conference & Exposition. 26–1238.
[83]
David Wood, Jerome S Bruner, and Gail Ross. 1976. The role of tutoring in problem solving. Journal of child psychology and psychiatry 17, 2 (1976), 89–100.
[84]
Zihan Wu. 2023. Investigating the Effectiveness of Variations of Micro Parsons Problems. In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 2. 120–122.
[85]
Zihan Wu, Barbara Ericson, and Christopher Brooks. 2021. Regex Parsons: Using Horizontal Parsons Problems to Scaffold Learning Regex. In Proceedings of the 21st Koli Calling International Conference on Computing Education Research. 1–3.
[86]
Zihan Wu and David H Smith IV. 2024. Evaluating Micro Parsons Problems as Exam Questions. arXiv preprint arXiv:2405.19460 (2024).
[87]
Benjamin Xie, Dastyni Loksa, Greg L Nelson, Matthew J Davidson, Dongsheng Dong, Harrison Kwik, Alex Hui Tan, Leanne Hwa, Min Li, and Amy J Ko. 2019. A theory of instruction for introductory programming skills. Computer Science Education 29, 2-3 (2019), 205–253.
[88]
Bin Zhao. 2011. Learning from errors: The role of context, emotion, and personality. Journal of organizational Behavior 32, 3 (2011), 435–463.
[89]
Craig B Zilles, Matthew West, Geoffrey L Herman, and Timothy Bretl. 2019. Every University Should Have a Computer-Based Testing Facility. In CSEDU (1). 414–420.
Index Terms
- Distractors Make You Pay Attention: Investigating the Learning Outcomes of Including Distractor Blocks in Parsons Problems
Recommendations
Using Adaptive Parsons Problems to Scaffold Write-Code Problems
ICER '22: Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1In this paper, we explore using Parsons problems to scaffold novice programmers who are struggling while solving write-code problems. Parsons problems, in which students put mixed-up code blocks in order, can be created quickly and already serve ...
Investigating the Role and Impact of Distractors on Parsons Problems in CS1 Assessments
ITiCSE 2023: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1In recent years Parsons problems have grown in popularity as both a pedagogical tool and as an assessment item alike. In these problems, students are expected to take existing but jumbled blocks of code and organize them to form a working solution. It is ...
Distractors in Parsons Problems Decrease Learning Efficiency for Young Novice Programmers
ICER '16: Proceedings of the 2016 ACM Conference on International Computing Education ResearchParsons problems are an increasingly popular method for helping inexperienced programmers improve their programming skills. In Parsons problems, learners are given a set of programming statements that they must assemble into the correct order. Parsons ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
August 2024
539 pages
ISBN:9798400704758
DOI:10.1145/3632620
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 12 August 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- National Science Foundation
Conference
ICER 2024
Sponsor:
ICER 2024: ACM Conference on International Computing Education Research
August 13 - 15, 2024
VIC, Melbourne, Australia
Acceptance Rates
Overall Acceptance Rate 189 of 803 submissions, 24%
Upcoming Conference
ICER 2025
- Sponsor:
- sigcse
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 105Total Downloads
- Downloads (Last 12 months)105
- Downloads (Last 6 weeks)55
Reflects downloads up to 20 Nov 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in