research-article

Using Fast Model-Based Fault Localisation to Aid Students in Self-Guided Program Repair and to Improve Assessment

Authors:

Michael PoppletonAuthors Info & Claims

ITiCSE '16: Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education

Pages 168 - 173

https://doi.org/10.1145/2899415.2899433

Published: 11 July 2016 Publication History

Abstract

Computer science instructors need to manage the rapid improvement of novice programmers through teaching, self-guided learning, and assessment. Appropriate feedback, both generic and personalised, is essential to facilitate student progress. Automated feedback tools can also accelerate the marking process and allow instructors to dedicate more time to other forms of tuition and students to progress more rapidly. Massive Open Online Courses rely on automated tools for both self-guided learning and assessment.

Fault localisation takes a significant part of debugging time. Popular spectrum-based methods do not narrow the potential fault locations sufficiently to assist novices. We therefore use a fast and precise model-based fault localisation method and show how it can be used to improve self-guided learning and accelerate assessment. We apply this to a large selection of actual student coursework submissions, providing more precise localisation within a sub-second response time. We show this using small test suites, already provided in the coursework management system, and on expanded test suites, demonstrating scaling. We also show compliance with test suites does not predictably score a class of "almost correct'' submissions, which our tool highlights.

References

[1]

P. Antonucci et al. An Incremental Hint System For Automated Programming Assignments. ITiCSE '15, pp. 320--325, 2015.

Digital Library

[2]

S. D. Benford et al. ACM Southeast Regional Conf., pp. 176--182, 1995.

Digital Library

[3]

G. Birch, B. Fischer, and M. R. Poppleton. Fast Model-Based Fault Localisation with Test Suites. TAP '15, LNCS 9154, pp. 38--57, 2015.

[4]

H. Blau and J. Eliot. FrenchPress Gives Students Automated Feedback on Java Program Flaws. ITiCSE '15, pp. 15--20, 2015.

Digital Library

[5]

K. Buffardi and S. H. Edwards. Exploring Influences on Student Adherence to Test-Driven Development. ITiCSE '12, pp. 105--110, 2012.

Digital Library

[6]

K. Buffardi and S. H. Edwards. Responses to Adaptive Feedback for Software Testing. ITiCSE '14, pp. 165--170, 2014.

Digital Library

[7]

J. deKleer and B. Williams. Diagnosing Multiple Faults. J. Artificial Intelligence, 32(1):97--130, 1987.

Digital Library

[8]

P. Denny, A. L. Reilly, and D. Carpenter. Enhancing Syntax Error Messages Appears Ineffectual. ITiCSE '14, pp. 273--278, 2014.

Digital Library

[9]

S. H. Edwards. Using Software Testing to Move Students from Trial-and-Error to Reflection-in-Action. SIGCSE '04, pp. 26--30, 2004.

Digital Library

[10]

A. Griesmayer, S. Staber, and R. Bloem. Automated Fault Localization for C Programs. Electronic Notes in Theoretical Computer Science, pp. 95--111, 2007.

Digital Library

[11]

Hawk-Eye: 2010. http://code.google.com/p/hawk-eye/.

[12]

M. T. Helmick. Interface-based Programming Assignments and Automatic Grading of Java Programs. ITiCSE '07, pp. 63--67, 2007.

Digital Library

[13]

D. Insa and J. Silva. Semi-Automatic Assessment of Unrestrained Java Code: A Library, a DSL, and a Workbench to Assess Exams and Exercises. ITiCSE '15, pp. 39--44, 2015.

Digital Library

[14]

D. Jackson and M. Usher. Grading Student Programs Using ASSYST. SIGCSE '97, pp. 335--339, 1997.

Digital Library

[15]

M. Vujošević-Jančić et al. Software Verification and Graph Similarity for Automated Evaluation of Students' Assignments. Inf. Softw. Technol., 55(6):1004--1016, 2013.

Digital Library

[16]

M. Joy, N. Griffiths, and R. Boyatt. The Boss Online Submission and Assessment System. J. Educational Resources in Computing, 5(3):2A, 2005.

Digital Library

[17]

M-J. Laakso et al. Automatic Assessment of Exercises for Algorithms and Data Structures - a Case Study with TRAKLA2. In Finnish/Baltic Sea Conf. on Comp. Sci. Edu., pp. 28--36, 2004.

[18]

R. Lister et al. Naturally Occurring Data As Research Instrument: Analyzing Examination Responses to Study the Novice Programmer. SIGCSE Bull., 41(4):156--173, 2010.

Digital Library

[19]

T. MacWilliam and D. J. Malan. Streamlining Grading Toward Better Feedback. ITiCSE '13, pp. 147--152, 2013.

Digital Library

[20]

L. Malmi, A. Korhonen, and R. Saikkonen. Experiences in Automatic Assessment on Mass Courses and Issues for Designing Virtual Courses. ITiCSE '02, pp. 55--59, 2002.

Digital Library

[21]

M. McCracken et al. A Multi-National, Multi-Institutional Study of Assessment of Programming Skills of First-year CS Students. ITiCSE-WGR '01, pp. 125--180, 2001.

Digital Library

[22]

C. Parnin and A. Orso. Are Automated Debugging Techniques Actually Helping Programmers? ISSTA '11, pp. 199--209, 2011.

Digital Library

[23]

D. N. Perkins et al. Conditions of Learning in Novice Programmers. J. Educational Computing Research, 2(1):37--55, 1986.

[24]

L. H. Pham et al. Assisting Students in Finding Bugs and their Locations in Programming Solutions. Int. J. Quality Ass. in Eng. and Tech. Edu., 3(2):12--27, 2014.

[25]

V. Pieterse. Automated Assessment of Programming Assignments. Comp. Sci. Edu. Research Conf., CSERC '13, pp. 45--56, 2013.

Digital Library

[26]

A. L. Reilly et al. On the Differences Between Correct Student Solutions. ITiCSE '13, pp. 177--182, 2013.

Digital Library

[27]

R. Reiter. A Theory of Diagnosis from First Principles. J. Artificial Intelligence, 32(1):57--95, 1987.

Digital Library

[28]

R. Saikkonen, L. Malmi, and A. Korhonen. Fully Automatic Assessment of Programming Exercises. ITiCSE '01, pp. 133--136, 2001.

Digital Library

[29]

R. Singh, S. Gulwani, and A. S. Lezama. Automated Feedback Generation for Introductory Programming Assignments. PLDI '13, pp. 15--26, 2013.

Digital Library

[30]

E. Soloway and J. C. Spohrer. Studying the Novice Programmer. 1988.

Digital Library

[31]

J. Spacco et al. Towards Improving Programming Habits to Create Better Computer Science Course Outcomes. ITiCSE '15, pp. 320--325, 2015.

Digital Library

[32]

M. Striewe, M. Balz, and M. Goedicke. A Flexible and Modular Software Architecture for Computer Aided Assessments and Automated Marking. CSEDU '09, pp. 54--61, 2009.

[33]

N. Tillmann et al. Teaching and Learning Programming and Software Engineering via Interactive Gaming. ICSE '13, pp. 1117--1126, 2013.

Digital Library

[34]

Q. Wang, C. Parnin, and A. Orso. Evaluating the Usefulness of IR-Based Fault Localization Techniques. ISSTA '15, pp. 1--11, 2015.

Digital Library

[35]

J. Whalley and N. Kasto. A Qualitative Think-Aloud Study of Novice Programmers' Code Writing Strategies. ITiCSE '15, pp. 320--325, 2015.

Digital Library

[36]

D. Wood, J. S. Bruner, and G. Ross. The Role of Tutoring in Problem Solving. J. Child Psychology and Psychiatry, 17(2):89--100, 1976.

[37]

X. Xie et al. A Theoretical Analysis of the Risk Evaluation Formulas for Spectrum-based Fault Localization. In TOSEM '13, 22(4):31A, 2013.

Digital Library

[38]

A. Zeller. Isolating Cause-Effect Chains with AskIgor. IWPC '03, pp. 296--297, 2003.

Digital Library

Cited By

Ventsel AHansson SRickberg MMadisson M(2023)Building Resilience Against Hostile Information Influence Activities: How a New Media Literacy Learning Platform Was Developed for the Estonian Defense ForcesArmed Forces & Society10.1177/0095327X231163265(0095327X2311632)Online publication date: 18-Apr-2023
https://doi.org/10.1177/0095327X231163265
Wolters BAngel HLepcha I(2023)Student perception and performance with self‐guided learning in soil science laboratory coursesNatural Sciences Education10.1002/nse2.2012052:2Online publication date: 5-Jul-2023
https://doi.org/10.1002/nse2.20120
Nakamura TKondo MKamei YUbayashi N(2022)Evaluating Automated Program Repair Techniques using Introductory Programming Course Datasets2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC54236.2022.00102(569-574)Online publication date: Jun-2022
https://doi.org/10.1109/COMPSAC54236.2022.00102
Show More Cited By

Index Terms

Using Fast Model-Based Fault Localisation to Aid Students in Self-Guided Program Repair and to Improve Assessment

Recommendations

Formative assessment in b-learning: effectively monitoring students learning
TEEM '14: Proceedings of the Second International Conference on Technological Ecosystems for Enhancing Multiculturality

One of the objectives of Higher Education is the development of learner autonomy so that students can become effective and efficient learners, with the capability for lifelong learning and for actively engage in the development of a knowledge society. ...
Assessment that matters: balancing reliability and learner-centered pedagogy in MOOC assessment
LAK '20: Proceedings of the Tenth International Conference on Learning Analytics & Knowledge

Learner-centered pedagogy highlights active learning and formative feedback. Instructors often incentivize learners to engage in such formative assessment activities by crediting their completion and score in the final grade, a pedagogical practice that ...
Online formative assessment in higher education: A review of the literature

As online and blended learning has become common place educational strategy in higher education, educators need to reconceptualise fundamental issues of teaching, learning and assessment in non traditional spaces. These issues include concepts such as ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ITiCSE '16: Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education

July 2016

394 pages

ISBN:9781450342315

DOI:10.1145/2899415

General Chairs:
Alison Clear
Eastern Institute of Technology, New Zealand
,
Ernesto Cuadros-Vargas
San Pablo Catholic University, Peru
,
Program Chairs:
Janet Carter
University of Kent, UK
,
Yvan Tupac
San Pablo Catholic University, Peru

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Bilkent University: Bilkent University
SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Engineering and Physical Sciences Research Council UK

Conference

ITiCSE '16

Sponsor:

Bilkent University
SIGCSE

ITiCSE '16: Innovation and Technology in Computer Science Education Conference 2016

July 11 - 13, 2016

Arequipa, Peru

Acceptance Rates

ITiCSE '16 Paper Acceptance Rate 56 of 147 submissions, 38%;

Overall Acceptance Rate 552 of 1,613 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
199
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)3

Reflects downloads up to 28 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ventsel AHansson SRickberg MMadisson M(2023)Building Resilience Against Hostile Information Influence Activities: How a New Media Literacy Learning Platform Was Developed for the Estonian Defense ForcesArmed Forces & Society10.1177/0095327X231163265(0095327X2311632)Online publication date: 18-Apr-2023
https://doi.org/10.1177/0095327X231163265
Wolters BAngel HLepcha I(2023)Student perception and performance with self‐guided learning in soil science laboratory coursesNatural Sciences Education10.1002/nse2.2012052:2Online publication date: 5-Jul-2023
https://doi.org/10.1002/nse2.20120
Nakamura TKondo MKamei YUbayashi N(2022)Evaluating Automated Program Repair Techniques using Introductory Programming Course Datasets2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC54236.2022.00102(569-574)Online publication date: Jun-2022
https://doi.org/10.1109/COMPSAC54236.2022.00102
Endres MSakkas GCosman BJhala RWeimer WZimmermann TLawall JMarinov D(2019)InFixProceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE.2019.00045(399-410)Online publication date: 10-Nov-2019
https://dl.acm.org/doi/10.1109/ASE.2019.00045
Birch GFischer BPoppleton M(2019)Fast test suite-driven model-based fault localisation with application to pinpointing defects in student programsSoftware and Systems Modeling (SoSyM)10.1007/s10270-017-0612-y18:1(445-471)Online publication date: 1-Feb-2019
https://dl.acm.org/doi/10.1007/s10270-017-0612-y
Xu ZMa SZhang XZhu SXu BChaudron MCrnkovic IChechik MHarman M(2018)Debugging with intelligence via probabilistic inferenceProceedings of the 40th International Conference on Software Engineering10.1145/3180155.3180237(1171-1181)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3180155.3180237
Yi JAhmed UKarkare ATan SRoychoudhury ABodden ESchäfer WDeursen AZisman A(2017)A feasibility study of using automated program repair for introductory programming assignmentsProceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering10.1145/3106237.3106262(740-751)Online publication date: 21-Aug-2017
https://dl.acm.org/doi/10.1145/3106237.3106262

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents