research-article

Public Access

Distractors in Parsons Problems Decrease Learning Efficiency for Young Novice Programmers

Authors:

Kyle James Harms,

Caitlin L. KelleherAuthors Info & Claims

ICER '16: Proceedings of the 2016 ACM Conference on International Computing Education Research

Pages 241 - 250

https://doi.org/10.1145/2960310.2960314

Published: 25 August 2016 Publication History

Abstract

Parsons problems are an increasingly popular method for helping inexperienced programmers improve their programming skills. In Parsons problems, learners are given a set of programming statements that they must assemble into the correct order. Parsons problems commonly use distractors, extra statements that are not part of the solution. Yet, little is known about the effect distractors have on a learner's ability to acquire new programming skills. We present a study comparing the effectiveness of learning programming from Parsons problems with and without distractors. The results suggest that distractors decrease learning efficiency. We found that distractor participants showed no difference in transfer task performance compared to those without distractors. However, the distractors increased learners cognitive load, decreased their success at completing Parsons problems by 26%, and increased learners' time on task by 14%.

References

[1]

Bridger, E.K. and Mecklinger, A. 2014. Errorful and errorless learning: The impact of cue-target constraint in learning from errors. Memory & Cognition. 42, 6 (Mar. 2014), 898--911.

[2]

Buffum, P.S. et al. 2015. A Practical Guide to Developing and Validating Computer Science Knowledge Assessments with Application to Middle School. Proceedings of the 46th ACM Technical Symposium on Computer Science Education (New York, NY, USA, 2015), 622--627.

Digital Library

[3]

Burnett, M. et al. 2010. Gender Differences and Programming Environments: Across Programming Populations. Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (New York, NY, USA, 2010), 28:1--28:10.

Digital Library

[4]

Butler, A.C. et al. 2007. The effect of type and timing of feedback on learning from multiple-choice tests. Journal of Experimental Psychology: Applied. 13, 4 (2007), 273--281.

[5]

Butler, A.C. et al. 2006. When additional multiple-choice lures aid versus hinder later memory. Applied Cognitive Psychology. 20, 7 (Nov. 2006), 941--956.

[6]

Butler, A.C. and Roediger, H.L. 2008. Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Memory & Cognition. 36, 3 (Apr. 2008), 604--616.

[7]

Cyr, A.-A. and Anderson, N.D. 2015. Mistakes as stepping stones: Effects of errors on episodic memory among younger and older adults. Journal of Experimental Psychology: Learning, Memory, and Cognition. 41, 3 (2015), 841--850.

[8]

Denny, P. et al. 2008. Evaluating a New Exam Question: Parsons Problems. Proceedings of the Fourth International Workshop on Computing Education Research (New York, NY, USA, 2008), 113--124.

Digital Library

[9]

Elley, W.B. 1966. The Role of Errors in Learning with Feedback. British Journal of Educational Psychology. 36, 3 (Nov. 1966), 296--300.

[10]

Ericson, B.J. et al. 2015. Analysis of Interactive Features Designed to Enhance Learning in an Ebook. Proceedings of the Eleventh Annual International Conference on International Computing Education Research (New York, NY, USA, 2015), 169--178.

Digital Library

[11]

van Gog, T. et al. 2012. Timing and Frequency of Mental Effort Measurement: Evidence in Favour of Repeated Measures. Applied Cognitive Psychology. 26, 6 (Nov. 2012), 833--839.

[12]

Gog, T. van and Paas, F. 2008. Instructional Efficiency: Revisiting the Original Construct in Educational Research. Educational Psychologist. 43, 1 (Jan. 2008), 16--26.

[13]

Große, C.S. and Renkl, A. 2007. Finding and fixing errors in worked examples: Can this foster learning outcomes? Learning and Instruction. 17, 6 (Dec. 2007), 612--634.

[14]

Haladyna, T.M. and Downing, S.M. 1988. Functional Distractors: Implications for Test-Item Writing and Test Design. (Apr. 1988).

[15]

Haladyna, T.M. and Downing, S.M. 1993. How Many Options is Enough for a Multiple-Choice Test Item? Educational and Psychological Measurement. 53, 4 (Dec. 1993), 999--1010.

[16]

Haladyna, T.M. and Downing, S.M. 1989. Validity of a Taxonomy of Multiple-Choice Item-Writing Rules. Applied Measurement in Education. 2, 1 (Jan. 1989), 51--78.

[17]

Harms, K.J. et al. 2015. Enabling independent learning of programming concepts through programming completion puzzles. 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) (Oct. 2015), 271--279.

[18]

Helminen, J. et al. 2013. How Do Students Solve Parsons Programming Problems? -- Execution-Based vs. Line-Based Feedback. Learning and Teaching in Computing and Engineering (LaTiCE), 2013 (Mar. 2013), 55--61.

Digital Library

[19]

Hour of Code: http://csedweek.org/.

[20]

Huelser, B.J. and Metcalfe, J. 2011. Making related errors facilitates learning, but learners do not know it. Memory & Cognition. 40, 4 (Dec. 2011), 514--527.

[21]

Ihantola, P. et al. 2013. How to Study Programming on Mobile Touch Devices: Interactive Python Code Exercises. Proceedings of the 13th Koli Calling International Conference on Computing Education Research (New York, NY, USA, 2013), 51--58.

Digital Library

[22]

Ihantola, P. and Karavirta, V. 2011. Two-dimensional parson's puzzles: The concept, tools, and first observations. Journal of Information Technology Education. 10, (2011).

[23]

K, H. et al. 2011. Does incorrect guessing impair fact learning? Journal of Educational Psychology. 103, 1 (2011), 48--59.

[24]

Kalyuga, S. et al. 1999. Managing split-attention and redundancy in multimedia instruction. Applied cognitive psychology. 13, 4 (1999), 351--371.

[25]

Karavirta, V. et al. 2012. A Mobile Learning Application for Parsons Problems with Automatic Feedback. Proceedings of the 12th Koli Calling International Conference on Computing Education Research (New York, NY, USA, 2012), 11--18.

Digital Library

[26]

Karpicke, J.D. 2012. Retrieval-Based Learning Active Retrieval Promotes Meaningful Learning. Current Directions in Psychological Science. 21, 3 (Jun. 2012), 157--163.

[27]

King, K.V. et al. 2004. The distractor rationale taxonomy: Enhancing multiple-choice items in reading and mathematics. Assessment Report. Pearson. (2004).

[28]

Kirk, R.E. 1996. Practical Significance: A Concept Whose Time Has Come. Educational and Psychological Measurement. 56, 5 (Oct. 1996), 746--759.

[29]

Kornell, N. et al. 2009. Unsuccessful retrieval attempts enhance subsequent learning. Journal of Experimental Psychology: Learning, Memory, and Cognition. 35, 4 (2009), 989--998.

[30]

Lee, M.J. and Ko, A.J. 2015. Comparing the Effectiveness of Online Learning Approaches on CS1 Learning Outcomes. Proceedings of the Eleventh Annual International Conference on International Computing Education Research (New York, NY, USA, 2015), 237--246.

Digital Library

[31]

Leppink, J. et al. 2013. Development of an instrument for measuring different types of cognitive load. Behavior Research Methods. 45, 4 (Apr. 2013), 1058--1072.

[32]

Leppink, J. et al. 2014. Effects of pairs of problems and examples on task performance and different types of cognitive load. Learning and Instruction. 30, (Apr. 2014), 32--42.

[33]

Lin, J. et al. 2010. Distractor rationale taxonomy: Diagnostic assessment of reading with ordered multiple-choice items. American Educational Research Association, Denver, CO. (2010).

[34]

Little, J.L. and Bjork, E.L. 2012. The persisting benefits of using multiple-choice tests as learning events. Proceedings of the 34th Annual Conference of the Cognitive Science Society (2012), 683--688.

[35]

Looking Glass: http://lookingglass.wustl.edu/.

[36]

Marsh, E. and Cantor, A. 2014. Learning from the Test: Dos and Don'ts for Using Multiple-Choice Tests. Integrating Cognitive Science with Innovative Teaching in STEM Disciplines. (Sep. 2014).

[37]

Marsh, E.J. et al. 2009. Memorial consequences of answering SAT II questions. Journal of Experimental Psychology: Applied. 15, 1 (2009), 1--11.

[38]

Marsh, E.J. et al. 2007. The memorial consequences of multiple-choice testing. Psychonomic Bulletin & Review. 14, 2 (Apr. 2007), 194--199.

[39]

Mkrtchyan, A. 2011. Distractor Quality Analyze In Multiple Choice Questions Based On Information Retrieval Model. EDULEARN11 Proceedings. (2011), 1624--1631.

[40]

Morrison, B.B. et al. 2014. Measuring Cognitive Load in Introductory CS: Adaptation of an Instrument. Proceedings of the Tenth Annual Conference on International Computing Education Research (New York, NY, USA, 2014), 131--138.

Digital Library

[41]

Morrison, B.B. et al. 2016. Subgoals Help Students Solve Parsons Problems. Proceedings of the 47th ACM Technical Symposium on Computing Science Education (New York, NY, USA, 2016), 42--47.

Digital Library

[42]

Muller, D. a. et al. 2008. Saying the wrong thing: improving learning with multimedia by including misconceptions. Journal of Computer Assisted Learning. 24, 2 (Apr. 2008), 144--155.

[43]

Paas, F. and Van Merrienboer, J. 1994. Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. Journal of Educational Psychology. 86, 1 (1994), 122--133.

[44]

Paas, F.G. 1992. Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology. 84, 4 (1992), 429--434.

[45]

Paas, F.G.W.C. et al. 1994. Measurement of cognitive load in instructional research. Perceptual and Motor Skills. 79, 1 (Aug. 1994), 419--430.

[46]

Paas, F.G.W.C. and Merriënboer, J.J.G.V. 1993. The Efficiency of Instructional Conditions: An Approach to Combine Mental Effort and Performance Measures. Human Factors: The Journal of the Human Factors and Ergonomics Society. 35, 4 (Dec. 1993), 737--743.

[47]

Parsons, D. and Haden, P. 2006. Parson's Programming Puzzles: A Fun and Effective Learning Tool for First Programming Courses. Proceedings of the 8th Australasian Conference on Computing Education - Volume 52 (Darlinghurst, Australia, Australia, 2006), 157--163.

Digital Library

[48]

Potts, R. and Shanks, D.R. 2014. The benefit of generating errors during learning. Journal of Experimental Psychology: General. 143, 2 (2014), 644--667.

[49]

Richland, L.E. et al. 2009. The pretesting effect: Do unsuccessful retrieval attempts enhance learning? Journal of Experimental Psychology: Applied. 15, 3 (2009), 243--257.

[50]

Rodriguez, M.C. et al. 2014. Distractor Functioning in Modified Items for Test Accessibility. SAGE Open. 4, 4 (Oct. 2014), 2158244014553586.

[51]

Roediger, H.L. and Karpicke, J.D. 2006. Test-Enhanced Learning Taking Memory Tests Improves Long-Term Retention. Psychological Science. 17, 3 (Mar. 2006), 249--255.

[52]

Roediger III, H.L. and Butler, A.C. 2011. The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences. 15, 1 (Jan. 2011), 20--27.

[53]

Roediger III, H.L. and Marsh, E.J. 2005. The Positive and Negative Consequences of Multiple-Choice Testing. Journal of Experimental Psychology: Learning, Memory, and Cognition. 31, 5 (2005), 1155--1159.

[54]

Scratch: https://scratch.mit.edu/.

[55]

Smith, A.M. et al. 2012. A Case Study of Expressively Constrainable Level Design Automation Tools for a Puzzle Game. Proceedings of the International Conference on the Foundations of Digital Games (New York, NY, USA, 2012), 156--163.

Digital Library

[56]

Sullivan, G.M. and Feinn, R. 2012. Using Effect Size or Why the P Value Is Not Enough. Journal of Graduate Medical Education. 4, 3 (Sep. 2012), 279--282.

[57]

Tarrant, M. et al. 2009. An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis. BMC Medical Education. 9, (2009), 40.

[58]

Tobias, S. et al. 2014. Game-Based Learning. Handbook of Research on Educational Communications and Technology. J.M. Spector et al., eds. Springer New York. 485--503.

[59]

Van Merriënboer, J.J.G. 1990. Strategies for Programming Instruction in High School: Program Completion vs. Program Generation. Journal of Educational Computing Research. 6, 3 (Jan. 1990), 265--285.

[60]

Van Merrienboer, J.J.G. and De Croock, M.B.M. 1992. Strategies for Computer-Based Programming Instruction: Program Completion Vs. Program Generation. Journal of Educational Computing Research. 8, 3 (Jan. 1992), 365--394.

[61]

Wang, M. et al. 2015. Using Feedback to Improve Learning: Differentiating between Correct and Erroneous Examples. 2015 International Symposium on Educational Technology (ISET) (Jul. 2015), 99--103.

[62]

Williams, E. 1949. Experimental Designs Balanced for the Estimation of Residual Effects of Treatments. Australian Journal of Chemistry. 2, 2 (Jan. 1949), 149--168.

Cited By

Hou XWu ZWang XEricson BJoyner DKim MWang XXia M(2024)CodeTailor: LLM-Powered Personalized Parsons Puzzles for Engaging Support While Learning ProgrammingProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3662032(51-62)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3662032
del Carpio Gutierrez ADenny PLuxton-Reilly AMonga MLonati VBarendsen ESheard JPaterson J(2024)Automating Personalized Parsons Problems with Customized Contexts and ConceptsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653568(688-694)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653568
Hou IMan OMettille SGutierrez SAngelikas KMacNeil S(2024)More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons ProblemsProceedings of the 26th Australasian Computing Education Conference10.1145/3636243.3636247(29-38)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1145/3636243.3636247
Show More Cited By

Index Terms

Distractors in Parsons Problems Decrease Learning Efficiency for Young Novice Programmers
1. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education

Recommendations

Investigating the Role and Impact of Distractors on Parsons Problems in CS1 Assessments
ITiCSE 2023: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1

In recent years Parsons problems have grown in popularity as both a pedagogical tool and as an assessment item alike. In these problems, students are expected to take existing but jumbled blocks of code and organize them to form a working solution. It is ...
Comparing the Impacts of Visually Grouped and Jumbled Distractors on Parsons Problems in CS1 Assessments
CompEd 2023: Proceedings of the ACM Conference on Global Computing Education Vol 1

Parsons problems are a commonly used problem type typically used in introductory computer science courses. They involve organizing blocks containing segments of code to form a program. These questions often use ''distractors'' which are plausible, but ...
Parsons Problems usage within a MOOC Pedagogy: (Abstract Only)
SIGCSE '18: Proceedings of the 49th ACM Technical Symposium on Computer Science Education

Parsons Problems is an effective Introductory Programming teaching tool that allows students to arrange code to form a functional program, reducing their cognitive load so they can focus on programming concepts. This demonstration shows how Parsons ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICER '16: Proceedings of the 2016 ACM Conference on International Computing Education Research

August 2016

310 pages

ISBN:9781450344494

DOI:10.1145/2960310

General Chairs:
Judy Sheard
Monash University, Australia
,
Josh Tenenberg
University of Washington Tacoma, USA
,
Donald Chinn
University of Washington Tacoma, USA
,
Brian Dorn
University of Nebraska at Omaha, USA

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 August 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ICER '16

Sponsor:

SIGCSE

ICER '16: International Computing Education Research Conference

September 8 - 12, 2016

VIC, Melbourne, Australia

Acceptance Rates

ICER '16 Paper Acceptance Rate 26 of 102 submissions, 25%;

Overall Acceptance Rate 189 of 803 submissions, 24%

Upcoming Conference

ICER 2025

Sponsor:
sigcse

ACM Conference on International Computing Education Research

August 3 - 6, 2025

Charlottesville , VA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

44
Total Citations
View Citations
913
Total Downloads

Downloads (Last 12 months)151
Downloads (Last 6 weeks)15

Reflects downloads up to 30 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hou XWu ZWang XEricson BJoyner DKim MWang XXia M(2024)CodeTailor: LLM-Powered Personalized Parsons Puzzles for Engaging Support While Learning ProgrammingProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3662032(51-62)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3662032
del Carpio Gutierrez ADenny PLuxton-Reilly AMonga MLonati VBarendsen ESheard JPaterson J(2024)Automating Personalized Parsons Problems with Customized Contexts and ConceptsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653568(688-694)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653568
Hou IMan OMettille SGutierrez SAngelikas KMacNeil S(2024)More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons ProblemsProceedings of the 26th Australasian Computing Education Conference10.1145/3636243.3636247(29-38)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1145/3636243.3636247
Smith DPoulsen SEmeka CWu ZHaynes-Magyar CZilles C(2024)Distractors Make You Pay Attention: Investigating the Learning Outcomes of Including Distractor Blocks in Parsons ProblemsProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671114(177-191)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671114
Tabarsi BReichert HLytle NCatete VBarnes T(2024)Scaffolding Novices: Analyzing When and How Parsons Problems Impact Novice Programming in an Integrated Science AssignmentProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671110(42-54)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671110
Haynes-Magyar CStephenson BStone JBattestilli LRebelsky SShoop L(2024)Neurodiverse Programmers and the Accessibility of Parsons Problems: An Exploratory Multiple-Case StudyProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630898(491-497)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630898
Poulsen SGertner YChen HCosman BWest MHerman GStephenson BStone JBattestilli LRebelsky SShoop L(2024)Disentangling the Learning Gains from Reading a Book Chapter and Completing Proof Blocks ProblemsProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630831(1056-1062)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630831
Reichert HTabarsi BPrice TBarnes T(2024)Jigsaw: A Tool for Decomposing and Planning Programming Problems2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC60511.2024.00034(236-247)Online publication date: 2-Sep-2024
https://doi.org/10.1109/VL/HCC60511.2024.00034
Ericson BPearce JRodger SCsizmadia AGarcia RGutierrez FLiaskos KPadiyath AScott MSmith DWarriem JZavaleta Bernuy ALaakso MMonga MSimon Sheard J(2023)Multi-Institutional Multi-National Studies of Parsons ProblemsProceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education10.1145/3623762.3633498(57-107)Online publication date: 22-Dec-2023
https://dl.acm.org/doi/10.1145/3623762.3633498
Wu ZEricson BBrooks CLaakso MMonga MSimon Sheard J(2023)Using Micro Parsons Problems to Scaffold the Learning of Regular ExpressionsProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 110.1145/3587102.3588853(457-463)Online publication date: 29-Jun-2023
https://dl.acm.org/doi/10.1145/3587102.3588853
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents