research-article

Adaptive Immediate Feedback Can Improve Novice Programming Engagement and Intention to Persist in Computer Science

Authors:

Thomas W. Price,

Tiffany BarnesAuthors Info & Claims

ICER '20: Proceedings of the 2020 ACM Conference on International Computing Education Research

Pages 194 - 203

https://doi.org/10.1145/3372782.3406264

Published: 07 August 2020 Publication History

Abstract

Prior work suggests that novice programmers are greatly impacted by the feedback provided by their programming environments. While some research has examined the impact of feedback on student learning in programming, there is no work (to our knowledge) that examines the impact of adaptive immediate feedback within programming environments on students' desire to persist in computer science (CS). In this paper, we integrate an adaptive immediate feedback (AIF) system into a block-based programming environment. Our AIF system is novel because it provides personalized positive and corrective feedback to students in real time as they work. In a controlled pilot study with novice high-school programmers, we show that our AIF system significantly increased students' intentions to persist in CS, and that students using AIF had greater engagement (as measured by their lower idle time) compared to students in the control condition. Further, we found evidence that the AIF system may improve student learning, as measured by student performance in a subsequent task without AIF. In interviews, students found the system fun and helpful, and reported feeling more focused and engaged. We hope this paper spurs more research on adaptive immediate feedback and the impact of programming environments on students' intentions to persist in CS.

References

[1]

Vincent Aleven, Ido Roll, Bruce M. McLaren, and Kenneth R. Koedinger. 2016. Help Helps, But Only So Much: Research on Help Seeking with Intelligent Tutoring Systems. International Journal of Artificial Intelligence in Education 26, 1 (2016), 1--19.

[2]

Joshua D Angrist and Jörn-Steffen Pischke. 2008. Mostly harmless econometrics: An empiricist's companion. Princeton university press.

[3]

Susan J Ashford. 1986. Feedback-seeking in individual adaptation: A resource perspective. Academy of Management journal 29, 3 (1986), 465--487.

[4]

Susan J Ashford, Ruth Blatt, and Don VandeWalle. 2003. Reflections on the looking glass: A review of research on feedback-seeking behavior in organizations. Journal of management 29, 6 (2003), 773--799.

[5]

Michael Ball. 2018. Lambda: An Autograder for ????????. Technical Report. Electrical Engineering and Computer Sciences University of California at Berkeley.

[6]

Lecia J Barker, Charlie McDowell, and Kimberly Kalahar. 2009. Exploring factors that influence computer science introductory course students to persist in the major. ACM Sigcse Bulletin 41, 1 (2009), 153--157.

Digital Library

[7]

Devon Barrow, Antonija Mitrovic, Stellan Ohlsson, and Michael Grimley. 2008. Assessing the impact of positive feedback in constraint-based tutors. In International Conference on Intelligent Tutoring Systems. Springer, 250--259.

Digital Library

[8]

Brett A Becker, Graham Glanville, Ricardo Iwashima, Claire McDonnell, Kyle Goslin, and Catherine Mooney. 2016. Effective compiler error message enhancement for novice programming students. Computer Science Education 26, 2--3 (2016), 148--175.

[9]

Brett A. Becker, Kyle Goslin, and Graham Glanville. 2018. The Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test. (2018).

[10]

Maureen Biggers, Anne Brauer, and Tuba Yilmaz. 2008. Student perceptions of computer science: a retention study comparing graduating seniors with cs leavers. ACM SIGCSE Bulletin 40, 1 (2008), 402--406.

Digital Library

[11]

Phyllis C Blumenfeld, Toni M Kempler, and Joseph S Krajcik. 2006. Motivation and cognitive engagement in learning environments. na.

[12]

Kristy Elizabeth Boyer, Robert Phillips, Michael D Wallis, Mladen A Vouk, and James C Lester. 2008. Learner characteristics and feedback in tutorial dialogue. In Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics, 53--61.

[13]

Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77--101.

[14]

J. Bruin. 2011 (accessed April 6, 2020). INTRODUCTION TO LINEAR MIXED MODELS. https://stats.idre.ucla.edu/stata/ado/analysis/

[15]

Erin Cech, Brian Rubineau, Susan Silbey, and Caroll Seron. 2011. Professional role confidence and gendered persistence in engineering. American Sociological Review 76, 5 (2011), 641--666.

[16]

AT Corbett and John R Anderson. 1989. Feedback timing and student control in the LISP Intelligent Tutoring System. In Proceedings of the Fourth International Conference on AI and Education. 64--72.

[17]

Albert Corbett and John R. Anderson. 2001. Locus of Feedback Control in Computer-Based Tutoring: Impact on Learning Rate, Achievement and Attitudes. In Proceedings of the SIGCHI Conference on Human Computer Interaction. 245--252.

[18]

Shelley J Correll. 2004. Constraints into preferences: Gender, status, and emerging career aspirations. American sociological review 69, 1 (2004), 93--113.

[19]

Peter J Denning and Edward E Gordon. 2015. A technician shortage. Commun. ACM 58, 3 (2015), 28--30.

Digital Library

[20]

Daantje Derks, Arjan ER Bos, and Jasper Von Grumbkow. 2008. Emoticons and online message interpretation. Social Science Computer Review 26, 3 (2008), 379--388.

Digital Library

[21]

Barbara Di Eugenio, Davide Fossati, Stellan Ohlsson, and David Cosejo. 2009. Towards explaining effective tutorial dialogues. In Annual Meeting of the Cognitive Science Society. 1430--1435.

[22]

Yihuan Dong, Samiha Marwan, Veronica Catete, Thomas W. Price, and Tiffany Barnes. 2019. Defining Tinkering Behavior in Open-ended Block-based Programming Assignments. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. ACM, 1204--1210.

Digital Library

[23]

Stephen H Edwards and Krishnan Panamalai Murali. 2017. CodeWorkout: short programming exercises with built-in data collection. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education. 188--193.

Digital Library

[24]

Ayelet Fishbach and Stacey R Finkelstein. 2012. How feedback influences persistence, disengagement, and change in goal pursuit. Goal-directed behavior (2012), 203--230.

[25]

Davide Fossati, Barbara Di Eugenio, STELLAN Ohlsson, Christopher Brown, and Lin Chen. 2015. Data driven automatic feedback generation in the iList intelligent tutoring system. Technology, Instruction, Cognition and Learning 10, 1 (2015), 5--26.

[26]

Luke Gusukuma, Austin Cory Bart, Dennis Kafura, and Jeremy Ernst. 2018. Misconception-driven feedback: Results from an experimental study. In Proceedings of the 2018 ACM Conference on International Computing Education Research. 160--168.

Digital Library

[27]

Luke Gusukuma, Dennis Kafura, and Austin Cory Bart. 2017. Authoring feedback for novice programmers in a block-based language. In 2017 IEEE Blocks and Beyond Workshop (B&B). IEEE, 37--40.

[28]

Ottar Hellevik. 2009. Linear versus logistic regression when the dependent variable is a dichotomy. Quality & Quantity 43, 1 (2009), 59--74.

[29]

David Hovemeyer and Jaime Spacco. 2013. CloudCoder: a web-based programming exercise system. Journal of Computing Sciences in Colleges 28, 3 (2013), 30--30.

Digital Library

[30]

Tony Jenkins. 2001. The motivation of students of programming. In Proceedings of the 6th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education, ITiCSE 2001, Canterbury, UK, June 25-27, 2001. 53--56.

Digital Library

[31]

Sandra Katz, David Allbritton, John Aronis, Christine Wilson, and Mary Lou Soffa. 2006. Gender, achievement, and persistence in an undergraduate computer science program. ACM SIGMIS Database: the DATABASE for Advances in Information Systems 37, 4 (2006), 42--57.

Digital Library

[32]

Rex B Kline. 2015. Principles and practice of structural equation modeling. Guilford publications.

[33]

Michael J Lee and Andrew J Ko. 2011. Personifying programming tool feedback improves novice programmers' learning. In Proceedings of the seventh international workshop on Computing education research. ACM, 109--116.

Digital Library

[34]

Mark R Lepper, Maria Woolverton, Donna L Mumme, and J Gurtner. 1993. Motivational techniques of expert human tutors: Lessons for the design of computerbased tutors. Computers as cognitive tools 1993 (1993), 75--105.

[35]

Colleen M Lewis, Ken Yasuhara, and Ruth E Anderson. 2011. Deciding to major in computer science: a grounded theory of students' self-assessment of ability. In Proceedings of the seventh international workshop on Computing education research. 3--10.

Digital Library

[36]

R Luckin et al. 2007. Beyond the code-and-count analysis of tutoring dialogues. Artificial intelligence in education: Building technology rich learning contexts that work, R. Luckin, KR Koedinger, and J. Greer, Eds. IOS Press (2007), 349--356.

[37]

Moira Maguire and Brid Delahunt. 2017. Doing a thematic analysis: A practical, step-by-step guide for learning and teaching scholars. AISHE-J: The All Ireland Journal of Teaching and Learning in Higher Education 9, 3 (2017).

[38]

Chris Martin, Janet Hughes, and John Richards. 2017. Designing engaging learning experiences in programming. In International Conference on Computer Supported Education. Springer, 221--245.

[39]

Samiha Marwan, Anay Dombe, and Thomas W. Price. 2020. Unproductive Helpseeking in Programming: What it is and How to Address it?. In The Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science (ITiCSE'20). ACM.

[40]

Samiha Marwan, Joseph Jay Williams, and Thomas W. Price. 2019. An Evaluation of the Impact of Automated Programming Hints on Performance and Learning. In Proceedings of the 2019 ACM Conference on International Computing Education Research. ACM, 61--70.

[41]

Samiha Marwan, Nicholas Lytle, Joseph Jay Williams, and Thomas W. Price. 2019. The Impact of Adding Textual Explanations to Next-step Hints in a Novice Programming Environment. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education. ACM, 520--526.

[42]

David M Merolla, Richard T Serpe, Sheldon Stryker, and P Wesley Schultz. 2012. Structural precursors to identity processes: The role of proximate social structures. Social Psychology Quarterly 75, 2 (2012), 149--172.

[43]

Antonija Mitrovic, Stellan Ohlsson, and Devon K Barrow. 2013. The effect of positive feedback in a constraint-based intelligent tutoring system. Computers & Education 60, 1 (2013), 264--272.

Digital Library

[44]

Roxana Moreno and Richard E Mayer. 2004. Personalized messages that promote science learning in virtual environments. Journal of educational Psychology 96, 1 (2004), 165.

[45]

Laurie Murphy, Gary Lewandowski, Renée McCauley, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: the good, the bad, and the quirky--a qualitative analysis of novices' strategies. ACM SIGCSE Bulletin 40, 1 (2008), 163--167.

Digital Library

[46]

Susanne Narciss and Katja Huth. 2004. How to design informative tutoring feedback for multimedia learning. Instructional design for multimedia learning 181195 (2004).

[47]

David J Nicol and Debra Macfarlane-Dick. 2006. Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in higher education 31, 2 (2006), 199--218.

[48]

Pete Nordquist. 2007. Providing accurate and timely feedback by automatically grading student programming labs. Journal of Computing Sciences in Colleges 23, 2 (2007), 16--23.

Digital Library

[49]

Helen J Parkin, Stuart Hepplestone, Graham Holden, Brian Irwin, and Louise Thorpe. 2012. A role for technology in enhancing students' engagement with feedback. Assessment & Evaluation in Higher Education 37, 8 (2012), 963--973.

[50]

Gary D Phye and Thomas Andre. 1989. Delayed retention effect: attention, perseveration, or both? Contemporary Educational Psychology 14, 2 (1989), 173--185.

[51]

Thomas W. Price and Tiffany Barnes. 2015. Comparing Textual and Block Interfaces in a Novice Programming Environment. In Proceedings of the International Computing Education Research Conference.

[52]

Thomas W. Price, Yihuan Dong, and Dragan Lipovac. 2017. iSnap: Towards Intelligent Tutoring in Novice Programming Environments. In Proceedings of the ACM Technical Symposium on Computer Science Education.

Digital Library

[53]

Thomas W. Price, Zhongxiu Liu, Veronica Catete, and Tiffany Barnes. 2017. Factors Influencing Students' Help-Seeking Behavior while Programming with Human and Computer Tutors. In Proceedings of the International Computing Education Research Conference.

Digital Library

[54]

Thomas W. Price, Rui Zhi, and Tiffany Barnes. 2017. Hint Generation Under Uncertainty: The Effect of Hint Quality on Help-Seeking Behavior. In Proceedings of the International Conference on Artificial Intelligence in Education.

[55]

Monica A Riordan. 2017. Emojis as tools for emotion work: Communicating affect in text messages. Journal of Language and Social Psychology 36, 5 (2017), 549--567.

[56]

Kelly Rivers and Kenneth R. Koedinger. 2017. Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor. International Journal of Artificial Intelligence in Education 27, 1 (2017), 37--64.

[57]

Mary Catherine Scheeler, Kathy L Ruhl, and James K McAfee. 2004. Providing performance feedback to teachers: A review. Teacher education and special education 27, 4 (2004), 396--407.

[58]

Valerie J Shute. 2008. Focus on formative feedback. Review of educational research 78, 1 (2008), 153--189.

[59]

Marieke Thurlings, Marjan Vermeulen, Theo Bastiaens, and Sjef Stijnen. 2013. Understanding feedback: A learning theory perspective. Educational Research Review 9 (2013), 1--15.

[60]

Jodie B Ullman and Peter M Bentler. 2003. Structural equation modeling. Handbook of psychology (2003), 607--634.

[61]

Paul Von Hippel. 2015. Linear vs. logistic probability models: Which is better, and when. Statistical Horizons (2015).

[62]

Wengran Wang, Rui Zhi, Alexandra Milliken, Nicholas Lytle, and Thomas W. Price. 2020. Crescendo: Engaging Students to Self-Paced Programming Practices. In Proceedings of the ACM Technical Symposium on Computer Science Education.

[63]

Wengran Wang, Rui Zhi, Alexandra Milliken, Nicholas Lytle, and Thomas W. Price. 2020. Crescendo: Engaging Students to Self-Paced Programming Practices. In To be published in the 51st ACM Technical Symposium on Computer Science Education (SIGCSE '20).

[64]

David Weintrop and Uri Wilensky. 2015. Using Commutative Assessments to Compare Conceptual Understanding in Blocks-based and Text-based Programs. In ICER, Vol. 15. 101--110.

Digital Library

[65]

Rui Zhi, Thomas W. Price, Nicholas Lytle, Yihuan Dong, and Tiffany Barnes. 2018. Reducing the State Space of Programming Problems through Data-Driven Feature Detection. In Educational Data Mining in Computer Science Education (CSEDM) Workshop@ EDM

Cited By

Mzwri KTurcsányi-Szabo M(2025)The Impact of Prompt Engineering and a Generative AI-Driven Tool on Autonomous Learning: A Case StudyEducation Sciences10.3390/educsci1502019915:2(199)Online publication date: 7-Feb-2025
https://doi.org/10.3390/educsci15020199
Lai CLin C(2025)Analysis of Learning Behaviors and Outcomes for Students with Different Knowledge Levels: A Case Study of Intelligent Tutoring System for Coding and Learning (ITS-CAL)Applied Sciences10.3390/app1504192215:4(1922)Online publication date: 12-Feb-2025
https://doi.org/10.3390/app15041922
Kennel KIshimaru SKüchemann SSteinert SKuhn JRuzika S(2025)Gaze-Based Prediction of Students’ Math Difficulties - A Time Dynamic Machine Learning Approach to Enable Early Individual AssistanceInternational Journal of Artificial Intelligence in Education10.1007/s40593-024-00447-5Online publication date: 22-Jan-2025
https://doi.org/10.1007/s40593-024-00447-5
Show More Cited By

Index Terms

Adaptive Immediate Feedback Can Improve Novice Programming Engagement and Intention to Persist in Computer Science

Recommendations

Aiding the Transition from Novice to Traditional Programming Environments
IDC '17: Proceedings of the 2017 Conference on Interaction Design and Children

Novice programming environments are increasingly popular as introductory tools for teaching programming. Many of these tools differ significantly from traditional programming environments and are successful in motivating novices and making it simple to ...
Adaptive feedback evaluation
DIWEB'06: Proceedings of the 5th WSEAS International Conference on Distance Learning and Web Engineering

Feedback is a powerful educational tool which would substantially improve the learning. Feedback would be integrated into Computer Adaptive Testing (CAT) systems to support the learner. This paper presents various types of adaptive feedback which depend ...
Novice-friendly multi-armed robotics programming
RoSE '19: Proceedings of the 2nd International Workshop on Robotics Software Engineering

Collaborative robots are being applied in a growing number of usage scenarios, but their adoption is slowed down by the high complexity of robot programming. As previous prototype studies have shown, block-based programming environments can enable ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICER '20: Proceedings of the 2020 ACM Conference on International Computing Education Research

August 2020

364 pages

ISBN:9781450370929

DOI:10.1145/3372782

General Chairs:
Anthony Robins
University of Otago, New Zealand
,
Adon Moskal
Otago Polytechnic, New Zealand
,
Amy J. Ko
University of Washington, United States
,
Renée McCauley
College of Charleston, United States
,
Program Chairs:
Anthony Robins
University of Otago, New Zealand
,
Amy J. Ko
University of Washington, United States

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 August 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ICER '20

Sponsor:

SIGCSE

ICER '20: International Computing Education Research Conference

August 1 - 5, 2020

Virtual Event, New Zealand

Acceptance Rates

Overall Acceptance Rate 189 of 803 submissions, 24%

Upcoming Conference

ICER 2025

Sponsor:
sigcse

ACM Conference on International Computing Education Research

August 3 - 6, 2025

Charlottesville , VA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

66
Total Citations
View Citations
1,691
Total Downloads

Downloads (Last 12 months)366
Downloads (Last 6 weeks)44

Reflects downloads up to 13 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mzwri KTurcsányi-Szabo M(2025)The Impact of Prompt Engineering and a Generative AI-Driven Tool on Autonomous Learning: A Case StudyEducation Sciences10.3390/educsci1502019915:2(199)Online publication date: 7-Feb-2025
https://doi.org/10.3390/educsci15020199
Lai CLin C(2025)Analysis of Learning Behaviors and Outcomes for Students with Different Knowledge Levels: A Case Study of Intelligent Tutoring System for Coding and Learning (ITS-CAL)Applied Sciences10.3390/app1504192215:4(1922)Online publication date: 12-Feb-2025
https://doi.org/10.3390/app15041922
Kennel KIshimaru SKüchemann SSteinert SKuhn JRuzika S(2025)Gaze-Based Prediction of Students’ Math Difficulties - A Time Dynamic Machine Learning Approach to Enable Early Individual AssistanceInternational Journal of Artificial Intelligence in Education10.1007/s40593-024-00447-5Online publication date: 22-Jan-2025
https://doi.org/10.1007/s40593-024-00447-5
Kovilpillai JSingh AHamdan AMcKenna KRaza F(2024)AI Enhanced Micro-Credentials for Efficiency and AccessibilityIntegrating Micro-Credentials With AI in Open Education10.4018/979-8-3693-5488-9.ch008(153-194)Online publication date: 6-Dec-2024
https://doi.org/10.4018/979-8-3693-5488-9.ch008
Marwan SChoquette NCateté VB. Morrison B(2024)Exploring Novices' Problem-Solving Strategies in Computing and Math DomainsProceedings of the 24th Koli Calling International Conference on Computing Education Research10.1145/3699538.3699557(1-8)Online publication date: 12-Nov-2024
https://dl.acm.org/doi/10.1145/3699538.3699557
Morazán MKempinski OHermans FBohrer R(2024)Using Computation Graphs to Explain Nondeterminism to StudentsProceedings of the 2024 ACM SIGPLAN International Symposium on SPLASH-E10.1145/3689493.3689978(23-33)Online publication date: 17-Oct-2024
https://dl.acm.org/doi/10.1145/3689493.3689978
Musabirov IZavaleta Bernuy AChen PLiut MWilliams J(2024)Opportunities for Adaptive Experiments to Enable Continuous Improvement in Computer Science EducationProceedings of the 26th Western Canadian Conference on Computing Education10.1145/3660650.3660659(1-7)Online publication date: 2-May-2024
https://dl.acm.org/doi/10.1145/3660650.3660659
Nguyen HStott NAllan VJoyner DKim MWang XXia M(2024)Comparing Feedback from Large Language Models and Instructors: Teaching Computer Science at ScaleProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664660(335-339)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3664660
Tabarsi BReichert HLytle NCatete VBarnes T(2024)Scaffolding Novices: Analyzing When and How Parsons Problems Impact Novice Programming in an Integrated Science AssignmentProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671110(42-54)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671110
Ko SStephens-Martinez K(2024)The Trees in the Forest: Characterizing Computing Students' Individual Help-Seeking ApproachesProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671099(343-358)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3632620.3671099
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten