research-article

An Open-Source, API-Based Framework for Assessing the Correctness of Code in CS50

Authors:

Jelle van Assema,

David J. MalanAuthors Info & Claims

ITiCSE '20: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education

Pages 487 - 492

https://doi.org/10.1145/3341525.3387417

Published: 15 June 2020 Publication History

Abstract

We present check50, an open-source, extensible tool for assessing the correctness of students' code that provides a simple, functional framework for writing checks as well as an easy-to-use API that abstracts away common tasks, among them compiling and running programs, providing their inputs, and checking their outputs. As a result, check50 has allowed us to provide students with immediate feedback on their progress as they complete an assignment while also facilitating automatic and consistent grading, allowing teaching staff to spend more time giving tailored, qualitative feedback. We have found, though, that since introducing check50 in 2012 in CS50 at Harvard, students have begun to perceive the course's programming assignments as more time-consuming and difficult than in years past. We speculate that the feedback that check50 provides prior to students' submission of each assignment has compelled students to spend more time debugging than they had in the past. At the same time, students' correctness scores are now higher than ever.

References

[1]

Kirsti M Ala-Mutka. 2005. A survey of automated assessment approaches for programming assignments. Computer science education 15, 2 (2005), 83--102.

[2]

CodeLab by Turing's Craft. 2020. https://www.turingscraft.com/

[3]

Julio C Caiza and Jose M Del Alamo. 2013. Programming assignments automatic grading: review of tools and implementations. In 7th International Technology, Education and Development Conference (INTED2013). 5691.

[4]

Kyle Dewey, Phillip Conrad, Michelle Craig, and Elena Morozova. 2017. Evaluating Test Suite Effectiveness and Assessing Student Code via Constraint Logic Programming. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '17). Association for Computing Machinery, New York, NY, USA, 317'322. https://doi.org/10.1145/3059009. 3059051

Digital Library

[5]

Stephen H. Edwards. 2004. Using Software Testing to Move Students from Trialand- Error to Reflection-in-Action. In Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education (SIGCSE '04). Association for Computing Machinery, New York, NY, USA, 26--30. https://doi.org/10.1145/ 971300.971312

Digital Library

[6]

Gradescope. 2020. https://www.gradescope.com/

[7]

Jessica B Hamrick. 2016. Creating and grading IPython/Jupyter notebook assignments with NbGrader. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education. 242--242.

Digital Library

[8]

Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Seppälä. 2010. Review of recent systems for automatic assessment of programming assignments. In Proceedings of the 10th Koli calling international conference on computing education research. 86--93.

Digital Library

[9]

Tiobe Index. 2018. https://www.tiobe.com/tiobe-index/.

[10]

Ville Karavirta, Ari Korhonen, and Lauri Malmi. 2006. On the use of resubmissions in automatic assessment systems. Computer science education 16, 3 (2006), 229-- 240.

[11]

OK. 2020. https://okpy.org/

[12]

Chris Piech, Jonathan Huang, Andy Nguyen, Mike Phulsuksombati, Mehran Sahami, and Leonidas J. Guibas. 2015. Learning Program Embeddings to Propagate Feedback on Student Code. CoRR abs/1505.05969 (2015). arXiv:1505.05969 http://arxiv.org/abs/1505.05969

[13]

Autolab Project. 2020. http://www.autolabproject.com/.

[14]

Riku Saikkonen, Lauri Malmi, and Ari Korhonen. 2001. Fully automatic assessment of programming exercises. In Proceedings of the 6th annual conference on Innovation and technology in computer science education. 133--136.

Digital Library

[15]

Rishabh Singh, Sumit Gulwani, and Armando Solar-Lezama. 2012. Automated Feedback Generation for Introductory Programming Assignments. arXiv:cs.PL/1204.1751

[16]

Vocareum. 2020. https://help.vocareum.com/article/16-grading-on-vocareum

Cited By

Cipriano BBaltazar BFachada NVourvopoulos AAlves P(2024)Bridging the Gap between Project-Oriented and Exercise-Oriented Automatic Assessment ToolsComputers10.3390/computers1307016213:7(162)Online publication date: 30-Jun-2024
https://doi.org/10.3390/computers13070162
Orvalho PJanota MManquinho VDorodchi MZhange MCooper S(2024)GitSEED: A Git-backed Automated Assessment Tool for Software Engineering and Programming EducationProceedings of the 2024 on ACM Virtual Global Computing Education Conference V. 110.1145/3649165.3690106(165-171)Online publication date: 5-Dec-2024
https://dl.acm.org/doi/10.1145/3649165.3690106
Messer MBrown NKölling MShi M(2024)Automated Grading and Feedback Tools for Programming Education: A Systematic ReviewACM Transactions on Computing Education10.1145/363651524:1(1-43)Online publication date: 19-Feb-2024
https://dl.acm.org/doi/10.1145/3636515
Show More Cited By

Index Terms

An Open-Source, API-Based Framework for Assessing the Correctness of Code in CS50
1. Applied computing
  1. Education
    1. Computer-assisted instruction
2. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education
      2. Student assessment

Recommendations

Mining api specifications from source code for improving software reliability
CS50 sandbox: secure execution of untrusted code
SIGCSE '13: Proceeding of the 44th ACM technical symposium on Computer science education

We introduce CS50 Sandbox, an environment for secure execution of untrusted code. Implemented as an asynchronous HTTP server, CS50 Sandbox offers clients the ability to execute programs (both interactive and non-interactive) written in any compiled or ...
Java(TM) Application Development on Linux(R) (Bruce Perens' Open Source Series)

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ITiCSE '20: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education

June 2020

615 pages

ISBN:9781450368742

DOI:10.1145/3341525

General Chairs:
Michail Giannakos
NTNU, Norway
,
Guttorm Sindre
Excited/NTNU, Norway
,
Program Chairs:
Andrew Luxton-Reilly
University of Auckland, NZ
,
Monica Divitini
NTNU, Norway

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 June 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ITiCSE '20

Sponsor:

SIGCSE

ITiCSE '20: Innovation and Technology in Computer Science Education

June 15 - 19, 2020

Trondheim, Norway

Acceptance Rates

Overall Acceptance Rate 552 of 1,613 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
219
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)3

Reflects downloads up to 29 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cipriano BBaltazar BFachada NVourvopoulos AAlves P(2024)Bridging the Gap between Project-Oriented and Exercise-Oriented Automatic Assessment ToolsComputers10.3390/computers1307016213:7(162)Online publication date: 30-Jun-2024
https://doi.org/10.3390/computers13070162
Orvalho PJanota MManquinho VDorodchi MZhange MCooper S(2024)GitSEED: A Git-backed Automated Assessment Tool for Software Engineering and Programming EducationProceedings of the 2024 on ACM Virtual Global Computing Education Conference V. 110.1145/3649165.3690106(165-171)Online publication date: 5-Dec-2024
https://dl.acm.org/doi/10.1145/3649165.3690106
Messer MBrown NKölling MShi M(2024)Automated Grading and Feedback Tools for Programming Education: A Systematic ReviewACM Transactions on Computing Education10.1145/363651524:1(1-43)Online publication date: 19-Feb-2024
https://dl.acm.org/doi/10.1145/3636515
Elhayany MMeinel CSpikol DViberg OMartínez-Monés AGuo P(2023)Towards Automated Code Assessment with OpenJupyter in MOOCsProceedings of the Tenth ACM Conference on Learning @ Scale10.1145/3573051.3596180(321-325)Online publication date: 20-Jul-2023
https://dl.acm.org/doi/10.1145/3573051.3596180
Mirhosseini SHenley AParnin CDoyle MStephenson BDorn BSoh LBattestilli L(2023)What Is Your Biggest Pain Point?Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 110.1145/3545945.3569816(291-297)Online publication date: 2-Mar-2023
https://dl.acm.org/doi/10.1145/3545945.3569816
Lau SEldridge JEllis SFraenkel ALanglois MRampure STiefenbruck JGuo PKizilcec RDavis KOchoa X(2022)The Challenges of Evolving Technical Courses at Scale: Four Case Studies of Updating Large Data Science CoursesProceedings of the Ninth ACM Conference on Learning @ Scale10.1145/3491140.3528278(201-211)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1145/3491140.3528278
Marchiori AMerkle LDoyle MSheard JSoh LDorn B(2022)LabtoolProceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 110.1145/3478431.3499285(1-7)Online publication date: 22-Feb-2022
https://dl.acm.org/doi/10.1145/3478431.3499285
Cipriano BFachada NAlves P(2022)Drop Project: An automatic assessment tool for programming assignmentsSoftwareX10.1016/j.softx.2022.10107918(101079)Online publication date: Jun-2022
https://doi.org/10.1016/j.softx.2022.101079
González-Carrillo CRestrepo-Calle FRamírez-Echeverry JGonzález F(2021)Automatic Grading Tool for Jupyter Notebooks in Artificial Intelligence CoursesSustainability10.3390/su13211205013:21(12050)Online publication date: 31-Oct-2021
https://doi.org/10.3390/su132112050
Annor PKayang EBoateng SBoateng G(2021)AutoGrad: Automated Grading Software for Mobile Game Assignments in SuaCode CoursesProceedings of the 10th Computer Science Education Research Conference10.1145/3507923.3507954(79-85)Online publication date: 22-Nov-2021
https://dl.acm.org/doi/10.1145/3507923.3507954

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents