research-article

Public Access

Misconception-Driven Feedback: Results from an Experimental Study

Authors:

Austin Cory Bart,

Jeremy ErnstAuthors Info & Claims

ICER '18: Proceedings of the 2018 ACM Conference on International Computing Education Research

Pages 160 - 168

https://doi.org/10.1145/3230977.3231002

Published: 08 August 2018 Publication History

Abstract

The feedback given to novice programmers can be substantially improved by delivering advice focused on learners' cognitive misconceptions contextualized to the instruction. Building on this idea, we present Misconception-Driven Feedback (MDF); MDF uses a cognitive student model and program analysis to detect mistakes and uncover underlying misconceptions. To evaluate the impact of MDF on student learning, we performed a quasi-experimental study of novice programmers that compares conventional run-time and output check feedback against MDF over three semesters. Inferential statistics indicates MDF supports significantly accelerated acquisition of conceptual knowledge and practical programming skills. Additionally, we present descriptive analysis from the study indicating the MDF student model allows for complex analysis of student mistakes and misconceptions that can suggest improvements to the feedback, the instruction, and to specific students.

References

[1]

John R Anderson, Daniel Bothell, Michael D Byrne, Scott Douglass, Christian Lebiere, and Yulin Qin. 2004. An integrated theory of the mind. Psychological review 111, 4 (2004), 1036.

[2]

John R Anderson, Frederick G Conrad, and Albert T Corbett. 1989. Skill acquisition and the LISP tutor. Cognitive Science 13, 4 (1989), 467--505.

[3]

Austin Cory Bart, Javier Tibau, Eli Tilevich, Clifford A Shaffer, and Dennis Kafura. 2017. BlockPy: An Open Access Data-Science Environment for Introductory Programmers. Computer 50, 5 (2017), 18--26.

Digital Library

[4]

Brett A Becker. 2015. An exploration of the effects of enhanced compiler error messages for computer programming novices. Master's thesis. Dublin Institute of Technology.

[5]

Benjamin S Bloom. 1984. The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational researcher 13, 6 (1984), 4--16.

[6]

Gary M Brosvic and Beth D Cohen. 1988. The horizontal-vertical illusion and knowledge of results. Perceptual and motor skills 67, 2 (1988), 463--469.

[7]

Ricardo Caceffo, Steve Wolfman, Kellogg S. Booth, and Rodolfo Azevedo. 2016. Developing a computer science concept inventory for introductory programming. In Proceedings of the 47th ACM Technical Symposium on Computer Science Education. ACM, 364--369.

Digital Library

[8]

Jacob Cohen. 1988. Statistical power analysis for the behavioral sciences. 2nd.

[9]

Albert T Corbett and John R Anderson. 2001. Locus of feedback control in computer-based tutoring: Impact on learning rate, achievement and attitudes. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 245--252.

Digital Library

[10]

Tyne Crow, Andrew Luxton-Reilly, and Burkhard Wuensche. 2018. Intelligent tutoring systems for programming education: a systematic review. In Proceedings of the 20th Australasian Computing Education Conference. ACM, 53--62.

Digital Library

[11]

Roberta E Dihoff, Gary M Brosvic, and Michael L Epstein. 2003. The role of feedback during academic testing: The delay retention effect revisited. The Psychological Record 53, 4 (2003), 533--548.

[12]

Luke Gusukuma, Austin Cory Bart, Dennis Kafura, Jeremy Ernst, and Katherine Cennamo. 2018. Instructional Design+ Knowledge Components: A Systematic Method for Refining Instruction. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education. ACM, 338--343.

Digital Library

[13]

Georgiana Haldeman, Andrew Tjang, Monica Babeş-Vroman, Stephen Bartos, Jay Shah, Danielle Yucht, and Thu D Nguyen. 2018. Providing Meaningful Feedback for Autograding of Programming Assignments. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education. ACM, 278--283.

Digital Library

[14]

Andrew Head, Elena Glassman, Gustavo Soares, Ryo Suzuki, Lucas Figueredo, Loris D'Antoni, and Björn Hartmann. 2017. Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis. In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale. ACM, 89--98.

Digital Library

[15]

Lisa Kaczmarczyk, Elizabeth Petrick, J Philip East, and Geoffrey L Herman. 2010. Identifying student misconceptions of programming. In Proceedings of the 41st ACM technical symposium on Computer science education. ACM, 107--111.

Digital Library

[16]

Dennis Kafura, Austin Cory Bart, and Bushra Chowdhury. 2015. Design and Preliminary Results From a Computational Thinking Course. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '15). ACM, 63--68.

Digital Library

[17]

Donald E Knuth. 1969. The art of computer programming. Vol. 1: Fundamental algorithms. Second printing.

[18]

Kenneth R Koedinger, Vincent Aleven, Neil Heffernan, Bruce McLaren, and Matthew Hockenberry. 2004. Opening the door to non-programmers: Authoring intelligent tutor behavior by demonstration. In International Conference on Intelligent Tutoring Systems. Springer, 162--174.

[19]

Kenneth R Koedinger, Albert T Corbett, and Charles Perfetti. 2012. The Knowledge-Learning-Instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive science 36, 5 (2012), 757--798.

[20]

Einari Kurvinen, Niko Hellgren, Erkki Kaila, Mikko-Jussi Laakso, and Tapio Salakoski. 2016. Programming misconceptions in an introductory level programming course exam. In Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education. ACM, 308--313.

Digital Library

[21]

Nguyen-Thinh Le. 2016. A classification of adaptive feedback in educational systems for programming. Systems 4, 2 (2016), 22.

[22]

David J Nicol and Debra Macfarlane-Dick. 2006. Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in higher education 31, 2 (2006), 199--218.

[23]

Chris Piech, Jonathan Huang, Andy Nguyen, Mike Phulsuksombati, Mehran Sahami, and Leonidas Guibas. 2015. Learning program embeddings to propagate feedback on student code. arXiv preprint arXiv:1505.05969 (2015).

Digital Library

[24]

Thomas W Price, Yihuan Dong, and Tiffany Barnes. 2016. Generating Data-driven Hints for Open-ended Programming. In EDM. 191--198.

[25]

Thomas W Price, Yihuan Dong, and Dragan Lipovac. 2017. iSnap: Towards Intelligent Tutoring in Novice Programming Environments. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, 483--488.

Digital Library

[26]

Kelly Rivers and Kenneth R Koedinger. 2017. Data-driven hint generation in vast solution spaces: a self-improving python programming tutor. International Journal of Artificial Intelligence in Education 27, 1 (2017), 37--64.

[27]

Takayuki Sekiya and Kazunori Yamaguchi. 2013. Tracing quiz set to identify novices' programming misconceptions. In Proceedings of the 13th Koli Calling International Conference on Computing Education Research. ACM, 87--95.

Digital Library

[28]

Valerie J Shute. 2008. Focus on formative feedback. Review of educational research 78, 1 (2008), 153--189.

[29]

Teemu Sirkiä and Juha Sorva. 2012. Exploring programming misconceptions: an analysis of student mistakes in visual program simulation exercises. In Proceedings of the 12th Koli Calling International Conference on Computing Education Research. ACM, 19--28.

Digital Library

[30]

Juha Sorva and Teemu Sirkiä. 2011. Context-sensitive guidance in the UUhistle program visualization system. In Proceedings of the 6th Program Visualization Workshop (PVW11). 77--85.

[31]

JC Spohrer and Elliot Soloway. 1986. Alternatives to construct-based programming misconceptions. In Acm sigchi bulletin, Vol. 17. ACM, 183--191.

Digital Library

[32]

Marieke Thurlings, Marjan Vermeulen, Theo Bastiaens, and Sjef Stijnen. 2013. Understanding feedback: A learning theory perspective. Educational Research Review 9 (2013), 1--15.

Cited By

Messer MBrown NKölling MShi M(2024)Automated Grading and Feedback Tools for Programming Education: A Systematic ReviewACM Transactions on Computing Education10.1145/363651524:1(1-43)Online publication date: 19-Feb-2024
https://dl.acm.org/doi/10.1145/3636515
Moons FHolvoet AKlingbeil KVandervieren E(2024)Comparing reusable, atomic feedback with classic feedback on a linear equations task using text mining and qualitative techniquesBritish Journal of Educational Technology10.1111/bjet.1344755:5(2257-2277)Online publication date: 26-Feb-2024
https://doi.org/10.1111/bjet.13447
Barros ANeto HCunha AMacedo NPaiva A(2024)Alloy Repair Hint Generation Based on Historical DataFormal Methods10.1007/978-3-031-71177-0_8(104-121)Online publication date: 13-Sep-2024
https://doi.org/10.1007/978-3-031-71177-0_8
Show More Cited By

Index Terms

Misconception-Driven Feedback: Results from an Experimental Study
1. Applied computing
  1. Education
    1. Learning management systems
2. Social and professional topics
  1. Professional topics
    1. Computing education

Recommendations

A Misconception Driven Student Model to Author Feedback
ICER '18: Proceedings of the 2018 ACM Conference on International Computing Education Research

Getting novice programmers over initial misconceptions is difficult because learning programming is difficult. Practice is one of the best ways for novices to learn. However, in the absence of feedback contextualized to instruction and focused on ...
BIOMIND Portal for Developing 21st Century Skills and Overcoming Students' Misconception in Biology Subject

This research was conducted to investigate the effectiveness of BIOMIND portal in enhancing students' 21st century skills and overcoming their misconceptions in Biology subject. 118 Indonesian high school students were involved in this quasi-...
Mobile Voting Systems for Creating Collaboration Environments and Getting Immediate Feedback: A New Curriculum Model of a University Lecture

Mobile devices can enhance learning and teaching by providing instant feedback and better diagnosis of learning problems, helping design new assessment models, enhancing learner autonomy and creating new formats of enquiry-based activities. The ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICER '18: Proceedings of the 2018 ACM Conference on International Computing Education Research

August 2018

307 pages

ISBN:9781450356282

DOI:10.1145/3230977

General Chairs:
Lauri Malmi
Aalto University, Finland
,
Ari Korhonen
Aalto University, Finland
,
Robert McCartney
University of Connecticut, USA
,
Andrew Petersen
University of Toronto Mississauga, Canada
,
Program Chairs:
Lauri Malmi
Aalto University, Finland
,
Robert McCartney
University of Connecticut, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 August 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

DUE
DGE

Conference

ICER '18

Sponsor:

SIGCSE

ICER '18: International Computing Education Research Conference

August 13 - 15, 2018

Espoo, Finland

Acceptance Rates

ICER '18 Paper Acceptance Rate 28 of 125 submissions, 22%;

Overall Acceptance Rate 189 of 803 submissions, 24%

Upcoming Conference

ICER 2025

Sponsor:
sigcse

ACM Conference on International Computing Education Research

August 3 - 6, 2025

Charlottesville , VA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

31
Total Citations
View Citations
1,228
Total Downloads

Downloads (Last 12 months)200
Downloads (Last 6 weeks)19

Reflects downloads up to 27 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Messer MBrown NKölling MShi M(2024)Automated Grading and Feedback Tools for Programming Education: A Systematic ReviewACM Transactions on Computing Education10.1145/363651524:1(1-43)Online publication date: 19-Feb-2024
https://dl.acm.org/doi/10.1145/3636515
Moons FHolvoet AKlingbeil KVandervieren E(2024)Comparing reusable, atomic feedback with classic feedback on a linear equations task using text mining and qualitative techniquesBritish Journal of Educational Technology10.1111/bjet.1344755:5(2257-2277)Online publication date: 26-Feb-2024
https://doi.org/10.1111/bjet.13447
Barros ANeto HCunha AMacedo NPaiva A(2024)Alloy Repair Hint Generation Based on Historical DataFormal Methods10.1007/978-3-031-71177-0_8(104-121)Online publication date: 13-Sep-2024
https://doi.org/10.1007/978-3-031-71177-0_8
Pan KJeffries BKoprinska I(2024)Predicting Successful Programming Submissions Based on Critical Logic BlocksArtificial Intelligence in Education10.1007/978-3-031-64299-9_32(363-371)Online publication date: 2-Jul-2024
https://doi.org/10.1007/978-3-031-64299-9_32
Sychev ODenisov M(2023)Explain Trace: Misconceptions of Control-Flow StatementsComputers10.3390/computers1210019212:10(192)Online publication date: 24-Sep-2023
https://doi.org/10.3390/computers12100192
Fischer BBirk FIwer EPanitz SDörner R(2023)Addressing Misconceptions in Introductory Programming: Automated Feedback in Integrated Development EnvironmentsProceedings of the 15th International Conference on Education Technology and Computers10.1145/3629296.3629297(1-8)Online publication date: 26-Sep-2023
https://dl.acm.org/doi/10.1145/3629296.3629297
Izu CMirolo CLaakso MMonga MSimon Sheard J(2023)Exploring CS1 Student's Notions of Code QualityProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 110.1145/3587102.3588808(12-18)Online publication date: 29-Jun-2023
https://dl.acm.org/doi/10.1145/3587102.3588808
Marwan SPrice T(2023)iSnap: Evolution and Evaluation of a Data-Driven Hint System for Block-Based ProgrammingIEEE Transactions on Learning Technologies10.1109/TLT.2022.322357716:3_Part_2(399-413)Online publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1109/TLT.2022.3223577
Deiner AFeldmeier PFraser GSchweikl SWang W(2023)Automated test generation for Scratch programsEmpirical Software Engineering10.1007/s10664-022-10255-x28:3Online publication date: 13-May-2023
https://dl.acm.org/doi/10.1007/s10664-022-10255-x
Du HXing WZhang Y(2022)Misconception of Abstraction: When to Use an Example and When to Use a Variable?Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 210.1145/3501709.3544276(28-29)Online publication date: 7-Aug-2022
https://dl.acm.org/doi/10.1145/3501709.3544276
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents