research-article

Comparing the Impacts of Visually Grouped and Jumbled Distractors on Parsons Problems in CS1 Assessments

Authors:

David H. Smith, IV,

Craig ZillesAuthors Info & Claims

CompEd 2023: Proceedings of the ACM Conference on Global Computing Education Vol 1

Pages 154 - 160

https://doi.org/10.1145/3576882.3617927

Published: 05 December 2023 Publication History

Abstract

Parsons problems are a commonly used problem type typically used in introductory computer science courses. They involve organizing blocks containing segments of code to form a program. These questions often use ''distractors'' which are plausible, but incorrect, blocks of code. In Parsons problems distractors are often included either by jumbling them in alongside the correct response options or visually grouping them with their correct alternatives. In this study, we investigate the impact of both jumbled and visually grouped distractors on: 1) student performance, 2) the amount of time students spent on the question, and 3) the item's quality in exams and quizzes in a CS1 Python course. Our findings indicate that the inclusion of distractors, both visually grouped and jumbled, have a marginal impact on item quality while increasing the amount of time needed to solve the problem and reducing students' performance. Though visually grouped distractors appear to mediate the amount of time students spend relative to their jumbled counterparts, their effect is not so large as to fully alleviate concerns related to the increase in duration.

References

[1]

Mary J Allen and Wendy M Yen. 2001. Introduction to measurement theory. Waveland Press.

[2]

Robert L Brennan. 1972. A generalized upper-lower item discrimination index. Educational and Psychological Measurement 32, 2 (1972), 289--303.

[3]

Derek C Briggs, Alicia C Alonzo, Cheryl Schwab, and Mark Wilson. 2006. Diagnostic assessment with ordered multiple-choice items. Educational Assessment 11, 1 (2006), 33--63.

[4]

Lee J Cronbach. 1951. Coefficient alpha and the internal structure of tests. psychometrika 16, 3 (1951), 297--334.

[5]

Paul Denny, Andrew Luxton-Reilly, and Beth Simon. 2008. Evaluating a new exam question: Parsons problems. In Proceedings of the fourth international workshop on computing education research. 113--124.

Digital Library

[6]

Yuemeng Du, Andrew Luxton-Reilly, and Paul Denny. 2020. A review of research on parsons problems. In Proceedings of the Twenty-Second Australasian Computing Education Conference. 195--202.

Digital Library

[7]

Robert L Ebel. 1967. The relation of item discrimination to test reliability. Journal of educational measurement 4, 3 (1967), 125--128.

[8]

Robert L Ebel and David A Frisbie. 1972. Essentials of educational measurement. Prentice-Hall Englewood Cliffs, NJ.

[9]

Max D Engelhart. 1965. A COMPARISON OF SEVERAL ITEM DISCRIMINATION INDICES 1. Journal of Educational Measurement 2, 1 (1965), 69--76.

[10]

Barbara Ericson, Austin McCall, and Kathryn Cunningham. 2019. Investigating the affect and effect of adaptive parsons problems. In Proceedings of the 19th Koli Calling International Conference on Computing Education Research. 1--10.

Digital Library

[11]

Barbara J Ericson, Paul Denny, James Prather, Rodrigo Duran, Arto Hellas, Juho Leinonen, Craig S Miller, Briana Morrison, Janice L Pearce, and Susan H Rodger. 2022a. Planning a Multi-institutional and Multi-national Study of the Effectiveness of Parsons Problems. In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 2. 576--577.

Digital Library

[12]

Barbara J Ericson, Paul Denny, James Prather, Rodrigo Duran, Arto Hellas, Juho Leinonen, Craig S Miller, Briana B Morrison, Janice L Pearce, and Susan H Rodger. 2022b. Parsons problems and beyond: Systematic literature review and empirical study designs. Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education (2022), 191--234.

Digital Library

[13]

Barbara J Ericson, James D Foley, and Jochen Rick. 2018. Evaluating the efficiency and effectiveness of adaptive parsons problems. In Proceedings of the 2018 ACM Conference on International Computing Education Research. 60--68.

Digital Library

[14]

Barbara J Ericson, Lauren E Margulieux, and Jochen Rick. 2017. Solving parsons problems versus fixing and writing code. In Proceedings of the 17th Koli Calling International Conference on Computing Education Research. 20--29.

Digital Library

[15]

Barbara J Ericson and Bradley N Miller. 2020. Runestone: A platform for free, on-line, and interactive ebooks. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. 1012--1018.

Digital Library

[16]

Barbara J Ericson, Janice L Pearce, Susan H Rodger, Andrew Csizmadia, Rita Garcia, Francisco J Gutierrez, Konstantinos Liaskos, Aadarsh Padiyath, Michael James Scott, David H Smith, et al. 2023. Conducting multi-institutional studies of Parsons problems. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2. 571--572.

Digital Library

[17]

Max Fowler, David H Smith IV, Mohammed Hassan, Seth Poulsen, Matthew West, and Craig Zilles. 2022. Reevaluating the relationship between explaining, tracing, and writing skills in CS1 in a replication study. Computer Science Education 32, 3 (2022), 355--383.

[18]

Flynn Fromont, Hiruna Jayamanne, and Paul Denny. 2023. Exploring the Difficulty of Faded Parsons Problems for Programming Education. In Proceedings of the 25th Australasian Computing Education Conference. 113--122.

Digital Library

[19]

Mark J Gierl, Okan Bulut, Qi Guo, and Xinxin Zhang. 2017. Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research 87, 6 (2017), 1082--1116.

[20]

Louis Guttman and Izchak M Schlesinger. 1967. Systematic construction of distractors for ability and achievement test items. Educational and Psychological Measurement 27, 3 (1967), 569--580.

[21]

Kyle James Harms, Jason Chen, and Caitlin L Kelleher. 2016. Distractors in Parsons problems decrease learning efficiency for young novice programmers. In Proceedings of the 2016 ACM Conference on International Computing Education Research. 241--250.

Digital Library

[22]

Carl C Haynes and Barbara J Ericson. 2021. Problem-solving efficiency and cognitive load for adaptive parsons problems vs. writing the equivalent code. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--15.

Digital Library

[23]

Angelica Hotiu. 2006. The relationship between item difficulty and discrimination indices in multiple-choice tests in a physical science course. Ph.,D. Dissertation. Citeseer.

[24]

Ville Karavirta, Juha Helminen, and Petri Ihantola. 2012. A mobile learning application for parsons problems with automatic feedback. In Proceedings of the 12th koli calling international conference on computing education research. 11--18.

Digital Library

[25]

Raymond Lister, Tony Clear, Dennis J Bouvier, Paul Carter, Anna Eckerdal, Jana Jacková, Mike Lopez, Robert McCartney, Phil Robbins, Otto Seppälä, et al. 2010. Naturally occurring data as research instrument: analyzing examination responses to study the novice programmer. ACM SIGCSE Bulletin 41, 4 (2010), 156--173.

Digital Library

[26]

Frederic M Lord. 1953. An application of confidence intervals and of maximum likelihood to the estimation of an examinee's ability. Psychometrika 18, 1 (1953), 57--76.

[27]

Frederic M Lord and Melvin R Novick. 2008. Statistical theories of mental test scores. IAP. 330--332 pages.

[28]

Wajiha Mahjabeen, Saeed Alam, Usman Hassan, Tahira Zafar, Rubab Butt, Sadaf Konain, and Myedah Rizvi. 2017. Difficulty index, discrimination index and distractor efficiency in multiple choice questions. Annals of PIMS-Shaheed Zulfiqar Ali Bhutto Medical University 13, 4 (2017), 310--315.

[29]

Bradley N. Miller and David L. Ranum. 2012. Beyond PDF and ePub: Toward an Interactive Textbook. In Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education (Haifa, Israel) (ITiCSE '12). ACM, New York, NY, USA, 150--155. https://doi.org/10.1145/2325296.2325335

Digital Library

[30]

Dale Parsons and Patricia Haden. 2006. Parson's programming puzzles: a fun and effective learning tool for first programming courses. In Proceedings of the 8th Australasian Conference on Computing Education-Volume 52. 157--163.

Digital Library

[31]

Seth Poulsen, Shubhang Kulkarni, Geoffrey Herman, and Matthew West. 2023. Efficient Feedback and Partial Credit Grading of Proof Blocks Problems. In Artificial Intelligence in Education: 24th International Conference, AIED 2023, Tokyo, Japan, July 3--7, 2023, Proceedings, Part I. Springer.

Digital Library

[32]

Adi Setiawan. 2014. Simulation study of item validity testing and item discrimination index. In 1st International Conference on Electrical Engineering, Computer Science and Informatics 2014. Institute of Advanced Engineering and Science.

[33]

David H Smith IV and Craig Zilles. 2023. Discovering, Autogenerating, and Evaluating Distractors for Python Parsons Problems in CS1. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. 924--930.

[34]

Tom AB Snijders and Roel J Bosker. 2011. Multilevel analysis: An introduction to basic and advanced multilevel modeling. Sage.

[35]

Rashmi Vyas and Avinash Supe. 2008. Multiple choice questions: a literature review on the optimal number of options. Natl Med J India 21, 3 (2008), 130--3.

[36]

Nathaniel Weinman, Armando Fox, and Marti Hearst. 2020. Exploring challenging variations of parsons problems. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. 1349--1349.

Digital Library

[37]

Nathaniel Weinman, Armando Fox, and Marti A Hearst. 2021. Improving instruction of programming patterns with faded parsons problems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--4.

Digital Library

[38]

Zihan Wu. 2023. Investigating the Effectiveness of Variations of Micro Parsons Problems. In Proceedings of the 2023 ACM Conference on International Computing Education Research-Volume 2. 120--122.

Digital Library

[39]

Zihan Wu, Barbara J Ericson, and Christopher Brooks. 2023. Using Micro Parsons Problems to Scaffold the Learning of Regular Expressions. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1. 457--463.

Digital Library

[40]

Craig B Zilles, Matthew West, Geoffrey L Herman, and Timothy Bretl. 2019. Every University Should Have a Computer-Based Testing Facility. In CSEDU (1). 414--420.

Cited By

Wu ZSmith DMonga MLonati VBarendsen ESheard JPaterson J(2024)Evaluating Micro Parsons Problems as Exam QuestionsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653583(674-680)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653583
Smith DPoulsen SEmeka CWu ZHaynes-Magyar CZilles C(2024)Distractors Make You Pay Attention: Investigating the Learning Outcomes of Including Distractor Blocks in Parsons ProblemsProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671114(177-191)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671114

Index Terms

Comparing the Impacts of Visually Grouped and Jumbled Distractors on Parsons Problems in CS1 Assessments
1. Social and professional topics
  1. Professional topics
    1. Computing education

Recommendations

Investigating the Role and Impact of Distractors on Parsons Problems in CS1 Assessments
ITiCSE 2023: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1

In recent years Parsons problems have grown in popularity as both a pedagogical tool and as an assessment item alike. In these problems, students are expected to take existing but jumbled blocks of code and organize them to form a working solution. It is ...
Discovering, Autogenerating, and Evaluating Distractors for Python Parsons Problems in CS1
SIGCSE 2023: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1

In this paper, we make three contributions related to the selection and use of distractors (lines of code reflecting common errors or misconceptions) in Parsons problems. First, we demonstrate a process by which templates for creating distractors can be ...
Distractors in Parsons Problems Decrease Learning Efficiency for Young Novice Programmers
ICER '16: Proceedings of the 2016 ACM Conference on International Computing Education Research

Parsons problems are an increasingly popular method for helping inexperienced programmers improve their programming skills. In Parsons problems, learners are given a set of programming statements that they must assemble into the correct order. Parsons ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CompEd 2023: Proceedings of the ACM Conference on Global Computing Education Vol 1

December 2023

180 pages

ISBN:9798400700484

DOI:10.1145/3576882

General Chairs:
Venkatesh Choppella
IIIT Hyderabad, India
,
Deepak B. Phatak
IIT Bombay, India
,
Program Chairs:
Andrew Luxton-Reilly
University of Auckland, New Zealand
,
Michelle Craig
University of Toronto, Canada

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 December 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CompEd 2023

Sponsor:

SIGCSE

CompEd 2023: ACM Global Computing Education Conference 2023

December 5 - 9, 2023

Hyderabad, India

Acceptance Rates

Overall Acceptance Rate 33 of 100 submissions, 33%

Upcoming Conference

CompEd '25

Sponsor:
sigcse

ACM Global Computing Education Conference 2025

October 21 - 25, 2025

Gaborone , Botswana

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
61
Total Downloads

Downloads (Last 12 months)61
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wu ZSmith DMonga MLonati VBarendsen ESheard JPaterson J(2024)Evaluating Micro Parsons Problems as Exam QuestionsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653583(674-680)Online publication date: 3-Jul-2024
https://dl.acm.org/doi/10.1145/3649217.3653583
Smith DPoulsen SEmeka CWu ZHaynes-Magyar CZilles C(2024)Distractors Make You Pay Attention: Investigating the Learning Outcomes of Including Distractor Blocks in Parsons ProblemsProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671114(177-191)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671114

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents