research-article

Scaling Expert Feedback: Two Case Studies

Author:

David A. JoynerAuthors Info & Claims

L@S '17: Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale

Pages 71 - 80

https://doi.org/10.1145/3051457.3051459

Published: 12 April 2017 Publication History

Abstract

Traditionally, education relies on a linear relationship between enrollment and staff; rising enrollment dictates increases to staff with some expertise (such as teaching assistants, TAs) for evaluation. This relationship is expensive, so learning at scale has largely deemphasized expert evaluation and feedback. Two organizations, though, have used different models to scale up class size online while retaining this expert evaluation and feedback. In this paper, we analyze the methods these two organizations have used to increase enrollment while preserving scalability and feedback. We observe an academic program has scaled feedback with traditional TAs by relying on unique characteristics of its student body, while a commercial program has done so with a novel, network-based model. These successes show the potential of learning from experts at scale.

References

[1]

Aleven, V., Mclaren, B. M., Sewall, J., & Koedinger, K. R. (2009). A new paradigm for intelligent tutoring systems: Example-tracing tutors. International Journal of Artificial Intelligence in Education, 19(2), 105--154.

Digital Library

[2]

Boud, D., Cohen, R., & Sampson, J. (2001). Peer Learning in Higher Education: Learning from & with Each Other. Psychology Press.

[3]

Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of educational research, 65(3), 245--281.

[4]

Carey, K. (2016, October 5). Georgia Tech's $7,000 Online Master's Degree Could Start a Revolution. New York Times, pp. A15.

[5]

Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70(3), 287--322.

[6]

Geigle, C., Zhai, C., & Ferguson, D. C. (2016, April). An Exploration of Automated Grading of Complex Assignments. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (pp. 351--360). ACM.

[7]

Goodman, J., Melkers, J., & Pallais, A. (2016). Can Online Delivery Increase Access to Education? HKS Faculty Research Working Paper Series RWP16-035.

[8]

Goel, A. & Joyner, D. A. (2016a). An Experiment in Teaching Cognitive Systems Online. In Haynes, D. (Ed.) International Journal for Scholarship of Technology-Enhanced Learning 1(1).

[9]

Goel, A. & Joyner, D. (2016b). Formative Assessment and Implicit Feedback in Online Learning. Presentation, Learning with MOOCs III.

[10]

Hext, J. B., & Winings, J. W. (1969). An automatic grading scheme for simple programming exercises. Communications of the ACM, 12(5), 272--275.

Digital Library

[11]

Jensen, J. C., Lee, E. A., & Seshia, S. A. (2013, April). Virtualizing cyber-physical systems: Bringing CPS to online education. In Proc. First Workshop on CPS Education (CPS-Ed).

[12]

Joyner, D. A., Ashby, W., Irish, L., Lam, Y., Langston, J., Lupiani, I., ... & Bruckman, A. (2016, April). Graders as Meta-Reviewers: Simultaneously Scaling and Improving Expert Evaluation for Large Online Classrooms. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (pp. 399--408). ACM.

[13]

Joyner, D. A., Goel, A., & Isbell, C. (2016). The Unexpected Pedagogical Benefits of Making Higher Education Accessible. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale. ACM.

Digital Library

[14]

Kendall, K., & Schussler, E. (2012). Does instructor type matter? Undergraduate student perception of graduate teaching assistants and professors. CBE-Life Sciences Education, 11(2), 187--199.

[15]

Kirschner, P., Sweller, J., & Clark, R. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75--86.

[16]

Kizilcec, R. F., & Schneider, E. (2015). Motivation as a lens to understand online learners: Toward datadriven design with the OLEI scale. ACM Transactions on Computer-Human Interaction (TOCHI), 22(2), 6.

Digital Library

[17]

Kolhe, P., Littman, M. L., & Isbell, C. L. (2016, April). Peer Reviewing Short Answers using Comparative Judgement. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (pp. 241--244). ACM.

[18]

Kulkarni, C., Bernstein, M. S., & Klemmer, S. (2015). PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance. In Proceedings from The Second ACM Conference on Learning @ Scale. ACM.

Digital Library

[19]

Lu, Y., Warren, J., Jermaine, C., Chaudhuri, S., & Rixner, S. (2015, May). Grading the Graders: Motivating Peer Graders in a MOOC. In Proceedings of the 24th International Conference on World Wide Web (pp. 680--690). ACM.

Digital Library

[20]

Lundstrom, K., & Baker, W. (2009). To give is better than to receive: The benefits of peer review to the reviewer's own writing. Journal of Second Language Writing, 18(1), 30--43.

[21]

Murray, T., Blessing, S., & Ainsworth, S. (2003). Authoring tools for advanced technology learning environments: Toward cost-effective adaptive, interactive and intelligent educational software. Springer Science & Business Media.

[22]

Polson, M. C., & Richardson, J. J. (2013). Foundations of intelligent tutoring systems. Psychology Press.

[23]

Raman, K., & Joachims, T. (2015). Bayesian Ordinal Peer Grading. In Proceedings from The Second ACM Conference on Learning @ Scale. ACM. 149--156.

Digital Library

[24]

Renkl, A. (2002). Worked-out examples: Instructional explanations support learning by self-explanations. Learning and Instruction, 12(5), 529--556.

[25]

Salzmann, C., Gillet, D., & Piguet, Y. (2016, February). MOOLs for MOOCs: A first edX scalable implementation. In 2016 13th International Conference on Remote Engineering and Virtual Instrumentation (REV) (pp. 246--251). IEEE.

[26]

Saraf, K. & Smith, C. (2016, October 5). "Coursera's superheroes: Meet the Mentor team." Coursera Blog. Retrieved from coursera.tumblr.com/post/151389966612/

[27]

Srikant, S., & Aggarwal, V. (2014, August). A system to grade computer programming skills using machine learning. In Proceedings of the 20th ACM International Conference on Knowledge Discovery and Data Mining (pp. 1887--1896). ACM.

Digital Library

[28]

VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227--265.

Digital Library

[29]

VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197--221.

[30]

Van Gog, T., & Rummel, N. (2010). Example-based learning: Integrating cognitive and social-cognitive research perspectives. Educational Psychology Review, 22(2), 155--174.

[31]

Van Zundert, M., Sluijsmans, D., & Van Merriënboer, J. (2010). Effective peer assessment processes: Research findings and future directions. Learning and Instruction, 20(4), 270--279.

[32]

Vogelsang, T., & Ruppertz, L. (2015, March). On the validity of peer grading and a cloud teaching assistant system. In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (pp. 41--50). ACM.

Digital Library

[33]

Yaron, D., Karabinos, M., Lange, D., Greeno, J. G., & Leinhardt, G. (2010). The ChemCollective-virtual labs for introductory chemistry courses. Science, 328(5978), 584--585.

Cited By

Duncan ATang THuang YLuu JThakkar NJoyner DJoyner DKim MWang XXia M(2024)Forums, Feedback, and Two Kinds of AI: A Selective History of Learning @ ScaleProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664667(376-382)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3664667
Adkins KJoyner DJoyner DKim MWang XXia M(2024)Newly Created Assignments and The First Repository Effect on Inter-Semester PlagiarismProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3662043(211-220)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3662043
Joyner DDuncan AJoyner DKim MWang XXia M(2024)Ten Years, Ten Trends: The First Decade of an Affordable At-Scale DegreeProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3662038(143-153)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3662038
Show More Cited By

Index Terms

Scaling Expert Feedback: Two Case Studies
1. Social and professional topics
  1. Professional topics
    1. Computing education
      1. Computing education programs
        Computer science education

Recommendations

Adapting the TPACK Framework for Online Teaching Within Higher Education

There are ongoing obstacles in accurately assessing and effectively applying TPACK within online higher education contexts. In this study, we adapt the original TPACK constructs for application within online higher education, and share the results of a ...
Scaling up for CS10K: Teaching and Supporting New Computer Science High School Teachers (Abstract Only)
SIGCSE '16: Proceedings of the 47th ACM Technical Symposium on Computing Science Education

Increasing need for computing expertise in our everyday lives and in the workforce, paired with declining enrollments in computing by women and underrepresented minorities have made it critical to provide students with experiences in computing before ...
Peer versus expert feedback: An investigation of the quality of peer feedback among secondary school students

Few studies have focused on peer assessment at the secondary school level. Consequently, we know very little about the quality of the feedback secondary school students can produce and its potential usefulness. This study was implemented in the context ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

L@S '17: Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale

April 2017

352 pages

ISBN:9781450344500

DOI:10.1145/3051457

General Chair:
Claudia Urrea
Massachusetts Institute of Technology, USA
,
Program Chairs:
Justin Reich
Massachusetts Institute of Technology, USA
,
Candace Thille
Stanford University, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

ACM: Association for Computing Machinery

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

L@S 2017

Sponsor:

ACM

L@S 2017: Fourth (2017) ACM Conference on Learning @ Scale

April 20 - 21, 2017

Massachusetts, Cambridge, USA

Acceptance Rates

L@S '17 Paper Acceptance Rate 14 of 105 submissions, 13%;

Overall Acceptance Rate 117 of 440 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
480
Total Downloads

Downloads (Last 12 months)39
Downloads (Last 6 weeks)3

Reflects downloads up to 18 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Duncan ATang THuang YLuu JThakkar NJoyner DJoyner DKim MWang XXia M(2024)Forums, Feedback, and Two Kinds of AI: A Selective History of Learning @ ScaleProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664667(376-382)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3664667
Adkins KJoyner DJoyner DKim MWang XXia M(2024)Newly Created Assignments and The First Repository Effect on Inter-Semester PlagiarismProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3662043(211-220)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3662043
Joyner DDuncan AJoyner DKim MWang XXia M(2024)Ten Years, Ten Trends: The First Decade of an Affordable At-Scale DegreeProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3662038(143-153)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3662038
Joyner DRusch ADuncan AWojcik JPopescu DSpikol DViberg OMartínez-Monés AGuo P(2023)Teaching at Scale and Back Again: The Impact of Instructors' Participation in At-Scale Education Initiatives on Traditional InstructionProceedings of the Tenth ACM Conference on Learning @ Scale10.1145/3573051.3593389(144-155)Online publication date: 20-Jul-2023
https://dl.acm.org/doi/10.1145/3573051.3593389
Eicher BJoyner D(2023)Making the Grade: Assessments at Scale in a Large Online Graduate Program2023 IEEE Learning with MOOCS (LWMOOCS)10.1109/LWMOOCS58322.2023.10305926(1-6)Online publication date: 11-Oct-2023
https://doi.org/10.1109/LWMOOCS58322.2023.10305926
Kasakowskij R(2022)Auswahl und Generierung von passenden Feedbacks auf Basis eines Feedback-Rating-System-FrameworksMedienPädagogik: Zeitschrift für Theorie und Praxis der Medienbildung10.21240/mpaed/48/2022.06.13.X48(155-169)Online publication date: 13-Jun-2022
https://doi.org/10.21240/mpaed/48/2022.06.13.X
Joyner DKizilcec RDavis KOchoa X(2022)Meet Me in the Middle: Retention in a "MOOC-Based" Degree ProgramProceedings of the Ninth ACM Conference on Learning @ Scale10.1145/3491140.3528283(82-92)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1145/3491140.3528283
Darvishi AKhosravi HAbdi SSadiq SGašević DKizilcec RDavis KOchoa X(2022)Incorporating Training, Self-monitoring and AI-Assistance to Improve Peer Feedback QualityProceedings of the Ninth ACM Conference on Learning @ Scale10.1145/3491140.3528265(35-47)Online publication date: 1-Jun-2022
https://dl.acm.org/doi/10.1145/3491140.3528265
Darvishi AKhosravi HSadiq SGašević D(2022) Incorporating AI and learning analytics to build trustworthy peer assessment systems British Journal of Educational Technology10.1111/bjet.1323353:4(844-875)Online publication date: 18-May-2022
https://doi.org/10.1111/bjet.13233
Tabebordbar ABeheshti ARezvani NAsadnia MElbourn S(2022)Resolving Learners Misunderstandings Using an Adaptive Divide and Correct TechniqueAdvances in Mobile Computing and Multimedia Intelligence10.1007/978-3-031-20436-4_15(154-168)Online publication date: 21-Nov-2022
https://doi.org/10.1007/978-3-031-20436-4_15
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents