short-paper

Improving Assessment on MOOCs Through Peer Identification and Aligned Incentives

Authors:

Dilrukshi Gamage,

Mark E. Whiting,

Thejan Rajapakshe,

Haritha Thilakarathne,

Shantha FernandoAuthors Info & Claims

L@S '17: Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale

Pages 315 - 318

https://doi.org/10.1145/3051457.3054013

Published: 12 April 2017 Publication History

Abstract

Massive Open Online Courses (MOOCs) use peer assessment to grade open ended questions at scale, allowing students to provide feedback. Relative to teacher based grading, peer assessment on MOOCs traditionally delivers lower quality feedback and fewer learner interactions. We present the identified peer review (IPR) framework, which provides non-blind peer assessment and incentives driving high quality feedback. We show that, compared to traditional peer assessment methods, IPR leads to significantly longer and more useful feedback as well as more discussion between peers.

References

[1]

Wilfried Admiraal, Bart Huisman, and Maarten Van de Ven. 2014. Self-and peer assessment in massive open online courses.InternationalJournal of Higher Education3, 3 (2014), p119.

[2]

Stephen P Balfour. 2013. Assessing writing in MOOCs: Automatedessay scoring and calibrated peer review (tm).Research & Practice inAssessment8 (2013).

[3]

Sian Bayne and Jen Ross. 2013. The Pedagogy of the Massive OpenOnline Course (MOOC): the UK View, the Higher Education Academy.(2013).

[4]

Stephen Bostock. 2000. Student peer assessment.Learning Technology(2000).{5}Snehalkumar Neil S Gaikwad, Durim Morina, Adam Ginzberg, Cather-ine Mullings, Shirish Goyal, Dilrukshi Gamage, Christopher Diemert,Mathias Burton, Sharon Zhou, Mark Whiting, and others. 2016.Boomerang: Rebounding the Consequences of Reputation Feedbackon Crowdsourcing Platforms. InProceedings of the 29th Annual Sym-posium on User Interface Software and Technology. ACM, 625--637.

Digital Library

[5]

Edward F Gehringer. 2000. Strategies and mechanisms for electronicpeer review. InFrontiers in Education Conference, 2000. FIE 2000. 30thAnnual, Vol. 1. IEEE, F1B--2.{7}John Hamer, Kenneth TK Ma, and Hugh HF Kwong. 2005. A methodof automatic grade calibration in peer assessment. InProceedings ofthe 7th Australasian conference on Computing education-Volume 42.Australian Computer Society, Inc., 67--72.

Digital Library

[6]

Soo-Min Kim, Patrick Pantel, Tim Chklovski, and Marco Pennacchiotti.2006. Automatically assessing review helpfulness. InProceedings of the2006 Conference on empirical methods in natural language processing.Association for Computational Linguistics, 423--430.

Digital Library

[7]

Avraham N Kluger and Angelo DeNisi. 1996. The effects of feedbackinterventions on performance: a historical review, a meta-analysis,and a preliminary feedback intervention theory.Psychological bulletin119, 2 (1996), 254.

[8]

Linda V Knight and Theresa A Steinbach. 2011. Adapting peer reviewto an online course: An exploratory case study.Journal of InformationTechnology Education10 (2011), 81--100.

[9]

Yasmine Kotturi, Chinmay E Kulkarni, Michael S Bernstein, and ScottKlemmer. 2015. Structure and messaging techniques for online peerlearning systems that increase stickiness. InProceedings of the Second(2015) ACM Conference on Learning@ Scale. ACM, 31--38.

Digital Library

[10]

Chinmay Kulkarni, Julia Cambre, Yasmine Kotturi, Michael S Bern-stein, and Scott R Klemmer. 2015. Talkabout: Making distance matterwith small groups in massive classes. InProceedings of the 18th ACMConference on Computer Supported Cooperative Work & Social Comput-ing. ACM, 1116--1128.

Digital Library

[11]

Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, KathrynPapadopoulos, Justin Cheng, Daphne Koller, and Scott R Klemmer.2015. Peer and self assessment in massive online classes. InDesignthinking research. Springer, 131--168.

[12]

Chinmay E Kulkarni, Michael S Bernstein, and Scott R Klemmer. 2015.PeerStudio: rapid peer feedback emphasizes revision and improvesperformance. InProceedings of the Second (2015) ACM Conference onLearning@ Scale. ACM, 75--84.

Digital Library

[13]

Ruiling Lu and Linda Bol. 2007. A comparison of anonymous versusidentiable e-peer review on college student writing performance andthe extent of critical feedback.Journal of Interactive Online Learning6, 2 (2007), 100--115.

[14]

Ken Reily, Pam Ludford Finnerty, and Loren Terveen. 2009. Twopeers are better than one: aggregating peer reviews for computingassignments is surprisingly accurate. InProceedings of the ACM 2009international conference on Supporting group work. ACM, 115--124.

Digital Library

[15]

Roger C Schank, Tamara R Berman, and Kimberli A Macpherson. 1999.Learning by doing.Instructional-design theories and models: A newparadigm of instructional theory2 (1999), 161--181.

[16]

George Siemens. 2005. Connectivism: Learning as network-creation.ASTD Learning News10, 1 (2005).

[17]

Thomas Staubitz, Dominic Petrick, Matthias Bauer, Jan Renz, andChristoph Meinel. 2016. Improving the Peer Assessment Experienceon MOOC Platforms. InProceedings of the Third (2016) ACM Conferenceon Learning@ Scale. ACM, 389--398.

Digital Library

[18]

Hoi K Suen. 2014. Peer assessment for massive open online courses(MOOCs).The International Review of Research in Open and DistributedLearning15, 3 (2014).{21}David Tinapple, Loren Olson, and John Sadauskas. 2013. CritViz: Web-based software supporting peer critique in large creative classrooms.Bulletin of the IEEE Technical Committee on Learning Technology15, 1(2013), 29.

[19]

Siri Vinther, O Haagen Nielsen, Jacob Rosenberg, Niels Keiding, andTV Shroeder. 2012. Same review quality in open versus blinded peerreview in" Ugeskrift for Læger.Dan Med J59, 8 (2012), A4479

Cited By

Sousa FAlves TGama SJorge JGonçalves D(2024)Studying how social relationships affect peer assessment in an E-learning environmentLearning Environments Research10.1007/s10984-024-09505-0Online publication date: 17-Jun-2024
https://doi.org/10.1007/s10984-024-09505-0
Adiguzel TKaya MBulut MMete SNalkesen-Akin Y(2023)Integrating cognitive presence strategies: A professional development training for K-12 teachersContemporary Educational Technology10.30935/cedtech/1298715:2(ep417)Online publication date: 2023
https://doi.org/10.30935/cedtech/12987
Popova TKolesova D(2023)The teaching model of mutual checking exercises in online course on Russian as a foreign languageRussian Language Studies10.22363/2618-8163-2023-21-2-181-19521:2(181-195)Online publication date: 15-Dec-2023
https://doi.org/10.22363/2618-8163-2023-21-2-181-195
Show More Cited By

Index Terms

Improving Assessment on MOOCs Through Peer Identification and Aligned Incentives

Recommendations

Improving the Peer Assessment Experience on MOOC Platforms
L@S '16: Proceedings of the Third (2016) ACM Conference on Learning @ Scale

Massive Open Online Courses (MOOCs) have revolutionized higher education by offering university-like courses for a large amount of learners via the Internet. The paper at hand takes a closer look on peer assessment as a tool for delivering ...
The Effect of Peer Assessment Rubrics on Learners' Satisfaction and Performance Within a Blended MOOC Environment
CSEDU 2015: Proceedings of the 7th International Conference on Computer Supported Education - Volume 2

Massive Open Online Courses (MOOCs) have a remarkable ability to expand access to a large scale of

participants worldwide, beyond the formality of the higher education systems. MOOCs support participants

to be actively involved in collaborative learning ...
Collaborative Calibrated Peer Assessment in Massive Open Online Courses

The free nature and open access courses in the Massive Open Online Courses MOOC allow the facilities of disseminating information for a large number of participants. However, the "massive" propriety can generate many pedagogical problems, such as the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

L@S '17: Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale

April 2017

352 pages

ISBN:9781450344500

DOI:10.1145/3051457

General Chair:
Claudia Urrea
Massachusetts Institute of Technology, USA
,
Program Chairs:
Justin Reich
Massachusetts Institute of Technology, USA
,
Candace Thille
Stanford University, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

L@S 2017

Sponsor:

ACM

L@S 2017: Fourth (2017) ACM Conference on Learning @ Scale

April 20 - 21, 2017

Massachusetts, Cambridge, USA

Acceptance Rates

L@S '17 Paper Acceptance Rate 14 of 105 submissions, 13%;

Overall Acceptance Rate 117 of 440 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
340
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)3

Reflects downloads up to 19 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sousa FAlves TGama SJorge JGonçalves D(2024)Studying how social relationships affect peer assessment in an E-learning environmentLearning Environments Research10.1007/s10984-024-09505-0Online publication date: 17-Jun-2024
https://doi.org/10.1007/s10984-024-09505-0
Adiguzel TKaya MBulut MMete SNalkesen-Akin Y(2023)Integrating cognitive presence strategies: A professional development training for K-12 teachersContemporary Educational Technology10.30935/cedtech/1298715:2(ep417)Online publication date: 2023
https://doi.org/10.30935/cedtech/12987
Popova TKolesova D(2023)The teaching model of mutual checking exercises in online course on Russian as a foreign languageRussian Language Studies10.22363/2618-8163-2023-21-2-181-19521:2(181-195)Online publication date: 15-Dec-2023
https://doi.org/10.22363/2618-8163-2023-21-2-181-195
Liu Jde Kok E(2023)CrowdLearn: Open Sourcing Learner Feedback to Enable Open-Ended Assessments in MOOCs2023 IEEE Learning with MOOCS (LWMOOCS)10.1109/LWMOOCS58322.2023.10306150(1-7)Online publication date: 11-Oct-2023
https://doi.org/10.1109/LWMOOCS58322.2023.10306150
Hertz BGrainger Clemson HTasic Hansen DLaurillard DMurray MFernandes LGilleran ARojas Ruiz DRutkauskiene D(2022)A pedagogical model for effective online teacher professional development—findings from the Teacher Academy initiative of the European CommissionEuropean Journal of Education10.1111/ejed.1248657:1(142-159)Online publication date: 19-Jan-2022
https://doi.org/10.1111/ejed.12486
Badea GPopescu E(2022)A dynamic review allocation approach for peer assessment in technology enhanced learningEducation and Information Technologies10.1007/s10639-022-11175-527:9(13131-13162)Online publication date: 27-Jun-2022
https://doi.org/10.1007/s10639-022-11175-5
Topping K(2021)Digital peer assessment in school teacher education and development: a systematic reviewResearch Papers in Education10.1080/02671522.2021.196130138:3(472-498)Online publication date: 15-Aug-2021
https://doi.org/10.1080/02671522.2021.1961301
Khatoon RJones E(2021)Flipped small group classes and peer marking: incentives, student participation and performance in a quasi-experimental approachAssessment & Evaluation in Higher Education10.1080/02602938.2021.198182347:6(910-927)Online publication date: 18-Oct-2021
https://doi.org/10.1080/02602938.2021.1981823
Yousef ASumner T(2020)Reflections on the last decade of MOOC researchComputer Applications in Engineering Education10.1002/cae.2233429:4(648-665)Online publication date: 30-Sep-2020
https://doi.org/10.1002/cae.22334
Popova TKolesova D(2019)Peer Assessment in Massive Open Online CoursesProceedings of the XI International Scientific Conference Communicative Strategies of the Information Society10.1145/3373722.3373794(1-5)Online publication date: 25-Oct-2019
https://dl.acm.org/doi/10.1145/3373722.3373794
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents