research-article

Using Peer Code Review as an Educational Tool

Authors:

Seth Copen Goldstein,

Majd SakrAuthors Info & Claims

ITiCSE '20: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education

Pages 173 - 179

https://doi.org/10.1145/3341525.3387370

Published: 15 June 2020 Publication History

Abstract

Code-review, the systematic examination of source code, is widely used in industry, but seldom used in courses. We designed and implemented a rubric-driven online peer code-review system (PCR) that we have deployed for two semesters, during which 228 students performed over 1003 code reviews. PCR is designed to meet four goals: (1) Provide timely feedback to students on their submissions, (2) Teach students the art of code review, (3) Allow custom feedback on submissions even in massive online classes, and (4) Allow students to learn from each other. We report on using PCR, in particular, the accuracy of student-based reviews, the surprising number of free-form comments made by students, the variability of staff-based reviews, how student engagement impacts the accuracy, the additional workload, and anecdotal perspectives of students. We describe some critical design considerations for PCR including rubric design, the importance of PCR training on each assignment to acclimate students to the rubric, and how we match student reviewers to student submissions.

References

[1]

Stephen P. Balfour. 2013. Assessing Writing in MOOCs: Automated Essay Scoring and Calibrated Peer Review?. Research & Practice in Assessment, Vol. 8 (2013), 40--48. https://www.learntechlib.org/p/157940

[2]

David Carless. 2006. Differing Perceptions in the Feedback Process. Studies in Higher Education, Vol. 31 (05 2006), 219--233. https://doi.org/10.1080/03075070600572132

[3]

codePost. 2020. Autograder and code review for computer science courses. www.codepost.io Download Jan 2020.

[4]

Christoffer Dall and Jason Nieh. 2014. Teaching Operating Systems Using Code Review. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (SIGCSE '14). Association for Computing Machinery, New York, NY, USA, 549--554. https://doi.org/10.1145/2538862.2538894

Digital Library

[5]

GitHub. 2020. GitHub. http://www.github.com/, last accessed in January 2020.

[6]

Elena L. Glassman, Jeremy Scott, Rishabh Singh, Philip J. Guo, and Robert C. Miller. 2015. OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale. ACM Trans. Comput.-Hum. Interact., Vol. 22, 2, Article Article 7 (March 2015), 35 pages. https://doi.org/10.1145/2699751

Digital Library

[7]

Gradescope. 2020. Gradescope. http://www.gradescope.com/, last accessed in January 2020.

[8]

Christopher Hundhausen, Anukrati Agrawal, Dana Fairbrother, and Michael Trevisan. 2009. Integrating Pedagogical Code Reviews into a CS 1 Course: An Empirical Study. SIGCSE Bull., Vol. 41, 1 (March 2009), 291--295. https://doi.org/10.1145/1539024.1508972

Digital Library

[9]

Open Learning Initiative. 2020. Open Learning Initiative. https://oli.cmu.edu, last accessed in January 2020.

[10]

Stephan Krusche, Mjellma Berisha, and Bernd Bruegge. 2016. Teaching Code Review Management Using Branch Based Workflows. In Proceedings of the 38th International Conference on Software Engineering Companion (ICSE '16). Association for Computing Machinery, New York, NY, USA, 384--393. https://doi.org/10.1145/2889160.2889191

Digital Library

[11]

Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R. Klemmer. 2013. Peer and self assessment in massive online classes. ACM Transactions on Computer-Human Interaction, Vol. 20, 6 (Jan 2013), 1--31. https://doi.org/10.1145/2505057

Digital Library

[12]

Heng Luo, Anthony Robinson, and Jae-Young Park. 2014. Peer Grading in a MOOC: Reliability, Validity, and Perceived Effects. Online Learning: Official Journal of the Online Learning Consortium, Vol. 18 (06 2014), 1--14. https://doi.org/10.24059/olj.v18i2.429

[13]

D. Milojicic. 2011. Autograding in the Cloud: Interview with David O'Hallaron. IEEE Internet Computing, Vol. 15, 1 (Jan 2011), 9--12. https://doi.org/10.1109/MIC.2011.2

Digital Library

[14]

Brian Railing. 2019. How Assignment Feedback goes unused. (2019). private communication.

[15]

Ralph Robinson. 2001. Calibrated Peer Review? an Application to Increase Student Reading & Writing Skills. The American Biology Teacher, Vol. 63 (09 2001), 474--480. https://doi.org/10.2307/4451167

[16]

Tom Trabasso and Gordon H. Bower. 1975. Attention in learning: theory and research .Robert E. Krieger, Huntington, NY.

[17]

Deborah A. Trytten. 2005. A Design for Team Peer Code Review. SIGCSE Bull., Vol. 37, 1 (Feb. 2005), 455--459. https://doi.org/10.1145/1047124.1047492

Digital Library

[18]

Carnegie Mellon Univesity. 2016. The Project Zone. An online LMS used for managing, among other things, programming projects.

[19]

Vocareum. 2020. Vocareum. http://www.vocareum.com/, last accessed in January 2020.

[20]

Yanqing Wang, Hang Li, Yuqiang Feng, Yu Jiang, and Ying Liu. 2012. Assessment of Programming Language Learning Based on Peer Code Review Model: Implementation and Experience Report. Comput. Educ., Vol. 59, 2 (Sept. 2012), 412--422. https://doi.org/10.1016/j.compedu.2012.01.007

[21]

Karl Eugene Wiegers. 2010. Peer reviews in software: a practical guide .Addison-Wesley, USA.

[22]

Andreas Zeller. 2000. Making Students Read and Review Code. In Proceedings of the 5th Annual SIGCSE/SIGCUE ITiCSE conference on Innovation and Technology in Computer Science Education (ITiCSE '00). Association for Computing Machinery, New York, NY, USA, 89--92. https://doi.org/10.1145/343048.343090

Digital Library

Cited By

Gudiño DFernández-Sánchez MBecerra-Traver MSánchez-Herrera S(2024)University Students’ Perceptions of Peer Assessment in Oral PresentationsEducation Sciences10.3390/educsci1403022114:3(221)Online publication date: 22-Feb-2024
https://doi.org/10.3390/educsci14030221
Zhang LChen TZong YGao XStephenson BStone JBattestilli LRebelsky SShoop L(2024)A Peer Grading Approach for Open-ended Programming Projects Based on Binary System and Swiss SystemProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630767(1484-1490)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630767
Middleton JOre JStolee KRoychoudhury APaiva AAbreu RStorey M(2024)Barriers for Students During Code Change ComprehensionProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639227(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639227
Show More Cited By

Index Terms

Using Peer Code Review as an Educational Tool
1. Applied computing
  1. Education
2. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education

Recommendations

A Review of Peer Code Review in Higher Education

Peer review is the standard process within academia for maintaining publication quality, but it is also widely employed in other settings, such as education and industry, for improving work quality and for generating actionable feedback to content ...
Recommending peer reviewers in modern code review: a multi-objective search-based approach
GECCO '20: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion

Modern code review is a common practice used by software developers to ensure high software quality in open source and industrial projects. During code review, developers submit their code changes which should be reviewed, via tool-based code review ...
Student attitudes and motivation for peer review in CS2
SIGCSE '11: Proceedings of the 42nd ACM technical symposium on Computer science education

Computer science students need experience with essential concepts and professional activities. Peer review is one way to meet these goals. In this work, we examine the students' attitudes towards and engagement in the peer review process, in early, ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ITiCSE '20: Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education

June 2020

615 pages

ISBN:9781450368742

DOI:10.1145/3341525

General Chairs:
Michail Giannakos
NTNU, Norway
,
Guttorm Sindre
Excited/NTNU, Norway
,
Program Chairs:
Andrew Luxton-Reilly
University of Auckland, NZ
,
Monica Divitini
NTNU, Norway

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCSE: ACM Special Interest Group on Computer Science Education

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 June 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

ITiCSE '20

Sponsor:

SIGCSE

ITiCSE '20: Innovation and Technology in Computer Science Education

June 15 - 19, 2020

Trondheim, Norway

Acceptance Rates

Overall Acceptance Rate 552 of 1,613 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
519
Total Downloads

Downloads (Last 12 months)83
Downloads (Last 6 weeks)6

Reflects downloads up to 27 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gudiño DFernández-Sánchez MBecerra-Traver MSánchez-Herrera S(2024)University Students’ Perceptions of Peer Assessment in Oral PresentationsEducation Sciences10.3390/educsci1403022114:3(221)Online publication date: 22-Feb-2024
https://doi.org/10.3390/educsci14030221
Zhang LChen TZong YGao XStephenson BStone JBattestilli LRebelsky SShoop L(2024)A Peer Grading Approach for Open-ended Programming Projects Based on Binary System and Swiss SystemProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630767(1484-1490)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3626252.3630767
Middleton JOre JStolee KRoychoudhury APaiva AAbreu RStorey M(2024)Barriers for Students During Code Change ComprehensionProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639227(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639227
Crandall ASprint GFischer B(2023)Generative Pre-Trained Transformer (GPT) Models as a Code Review Feedback Tool in Computer Science ProgramsJournal of Computing Sciences in Colleges10.5555/3636517.363652239:1(38-47)Online publication date: 6-Dec-2023
https://dl.acm.org/doi/10.5555/3636517.3636522
Braga PHébert KHudgins EScott EEdwards BSánchez Reyes LGrainger MForoughirad VHillemann FBinley ABrookson CGaynor KShafiei Sabet SGüncan AWeierbach HGomes DCrystal‐Ornelas R(2023) Not just for programmers: How GitHub can accelerate collaborative and reproducible research in ecology and evolution Methods in Ecology and Evolution10.1111/2041-210X.1410814:6(1364-1380)Online publication date: 21-Apr-2023
https://doi.org/10.1111/2041-210X.14108
Zhang LZong YChen TMa YAn SZhang L(2023)A Peer Review Approach to Grading Projects in Computer Courses2023 IEEE Frontiers in Education Conference (FIE)10.1109/FIE58773.2023.10343176(1-7)Online publication date: 18-Oct-2023
https://doi.org/10.1109/FIE58773.2023.10343176
Farah JSpaenlehauer BRodríguez-Triana MIngram SGillet D(2023)Integrating Code Reviews into Online Lessons to Support Software Engineering EducationLearning in the Age of Digital and Green Transition10.1007/978-3-031-26190-9_84(815-826)Online publication date: 23-Feb-2023
https://doi.org/10.1007/978-3-031-26190-9_84
Ada MMajid M(2022)Developing a system to increase motivation and engagement in student code peer review2022 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE)10.1109/TALE54877.2022.00023(93-98)Online publication date: Dec-2022
https://doi.org/10.1109/TALE54877.2022.00023
Farah JSpaenlehauer BRodriguez-Triana MIngram SGillet D(2022)Toward Code Review Notebooks2022 International Conference on Advanced Learning Technologies (ICALT)10.1109/ICALT55010.2022.00068(209-211)Online publication date: Jul-2022
https://doi.org/10.1109/ICALT55010.2022.00068
Indriasari TLuxton-Reilly ADenny P(2021)Improving Student Peer Code Review Using GamificationProceedings of the 23rd Australasian Computing Education Conference10.1145/3441636.3442308(80-87)Online publication date: 2-Feb-2021
https://dl.acm.org/doi/10.1145/3441636.3442308
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents