research-article

High-Resolution Course Feedback: Timely Feedback Mechanism for Instructors

Authors:

Chris PiechAuthors Info & Claims

L@S '23: Proceedings of the Tenth ACM Conference on Learning @ Scale

Pages 81 - 91

https://doi.org/10.1145/3573051.3593391

Published: 20 July 2023 Publication History

Abstract

We study the problem of minimizing the delay between when an issue comes up in a course and when instructors get feedback about it. The widespread practice of obtaining midterm and end-of-term feedback from students is suboptimal in this regard, especially for large courses: it over-samples at a specific point in the course and can be biased by factors irrelevant to the teaching process. As a solution, we release High Resolution Course Feedback (HRCF), an open-source student feedback mechanism that builds on a surprisingly simple idea: survey each student on random weeks exactly twice per term. Despite the simplicity of its core idea, when deployed to 31 courses totaling a cumulative 6,835 students, HRCF was able to detect meaningful mood changes in courses and significantly improve timely feedback without asking for extra work from students compared to the common practice. An interview with the instructors revealed that HRCF provided constructive and useful feedback about their courses early enough to be acted upon, which would have otherwise been unobtainable through other survey methods. We also explore the possibility of using Large Language Models to flexibly and intuitively organize large volumes of student feedback at scale and discuss how HRCF can be further improved.

References

[1]

Abbott, R. D., Wulff, D. H., Nyqist, J. D., Ropp, V. A., and Hess, C. W. Satisfaction with processes of collecting student opinions about instruction: The student perspective. Journal of Educational Psychology 82, 2 (1990), 201.

[2]

Abrami, P. C., d'Apollonia, S., and Cohen, P. A. Validity of student ratings of instruction: What we know and what we do not. Journal of educational psychology 82, 2 (1990), 219.

[3]

Adams, M. J., and Umbach, P. D. Nonresponse and online student evaluations of teaching: Understanding the influence of salience, fatigue, and academic environments. Research in Higher Education 53, 5 (2012), 576--591.

[4]

Arthur, L. From performativity to professionalism: Lecturers' responses to student feedback. Teaching in Higher Education 14, 4 (2009), 441--454.

[5]

Avery, R. J., Bryant, W. K., Mathios, A., Kang, H., and Bell, D. Electronic course evaluations: does an online delivery system influence student evaluations? The Journal of Economic Education 37, 1 (2006), 21--37.

[6]

Brown, M. J. Student perceptions of teaching evaluations. Journal of Instructional Psychology 35, 2 (2008), 177--182.

[7]

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877--1901.

[8]

Centra, J. A. Will teachers receive higher student evaluations by giving higher grades and less course work? Research in higher education 44, 5 (2003), 495--518.

[9]

Chen, Y., and Hoshower, L. B. Student evaluation of teaching effectiveness: An assessment of student perception and motivation. Assessment & evaluation in higher education 28, 1 (2003), 71--88.

[10]

Cohen, P. A. Student ratings of instruction and student achievement: A metaanalysis of multisection validity studies. Review of educational Research 51, 3 (1981), 281--309.

[11]

Cross, K. P., and Angelo, T. A. Classroom assessment techniques. a handbook for faculty.

[12]

Diamond, M. R. The usefulness of structured mid-term feedback as a catalyst for change in higher education classes. Active Learning in Higher Education 5, 3 (2004), 217--231.

[13]

Driscoll, L. A., and Goodwin, W. L. The effects of varying information about use and disposition of results on university students' evaluations of faculty and courses. American Educational Research Journal 16, 1 (1979), 25--37.

[14]

Ethier, S. N. The doctrine of chances: probabilistic aspects of gambling. Springer.

[15]

Feldman, K. A. College students' views of male and female college teachers: Part i-evidence from the social laboratory and experiments. Research in Higher Education 33 (1992), 317--375.

[16]

Flodén, J. The impact of student feedback on teaching in higher education. Assessment & Evaluation in Higher Education 42, 7 (2017), 1054--1068.

[17]

Gravestock, P., and Gregor-Greenleaf, E. Student course evaluations: Research, models and trends. Higher Education Quality Council of Ontario Toronto, 2008.

[18]

Greenwald, A. G., and Gillmore, G. M. Grading leniency is a removable contaminant of student ratings. American psychologist 52, 11 (1997), 1209.

[19]

Grönberg, N., Knutas, A., Hynninen, T., and Hujala, M. Palaute: An online text mining tool for analyzing written student course feedback. IEEE Access 9 (2021), 134518--134529.

[20]

Groves, R. M., and Peytcheva, E. The impact of nonresponse rates on nonresponse bias: a meta-analysis. Public opinion quarterly 72, 2 (2008), 167--189.

[21]

Hu, Y., Zhang, S., Sathy, V., Panter, A., and Bansal, M. Setsum: Summarization and visualization of student evaluations of teaching. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: System Demonstrations (2022), pp. 71-- 89.

[22]

Iarossi, G. The power of survey design: A user's guide for managing surveys, interpreting results, and influencing respondents. World Bank Publications, 2006.

[23]

Irons, A., and Elkington, S. Enhancing learning through formative assessment and feedback. Routledge, 2021.

[24]

Keutzer, C. S. Midterm evaluation of teaching provides helpful feedback to instructors. Teaching of psychology 20, 4 (1993), 238--240.

[25]

Kulik, J. A. Student ratings: Validity, utility, and controversy. New directions for institutional research 2001, 109 (2001), 9--25.

[26]

Lewis, K. G. Using midsemester student feedback and responding to it. New Directions for Teaching and Learning 2001, 87 (2001), 33--44.

[27]

Marsh, H. W. Students' evaluations of university teaching: Dimensionality, reliability, validity, potential biases and usefulness. The scholarship of teaching and learning in higher education: An evidence-based perspective (2007), 319--383.

[28]

Marsh, H.W., and Roche, L. The use of students' evaluations and an individually structured intervention to enhance university teaching effectiveness. American educational research journal 30, 1 (1993), 217--251.

[29]

Marsh, H. W., and Roche, L. A. Making students' evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American psychologist 52, 11 (1997), 1187.

[30]

McLaughlin, M. W., and Pfeifer, R. S. Teacher Evaluation: Improvement, Accountability, and Effective Learning. Teachers College Press, 1988.

[31]

Murray, H. G. Does evaluation of teaching lead to improvement of teaching? The International Journal for Academic Development 2, 1 (1997), 8--23.

[32]

Narasimhan, K. Improving the climate of teaching sessions: the use of evaluations by students and instructors. Quality in Higher Education 7, 3 (2001), 179--190.

[33]

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 (2022).

[34]

Overall, J., and Marsh, H.W. Midterm feedback from students: Its relationship to instructional improvement and students' cognitive and affective outcomes. Journal of educational psychology 71, 6 (1979), 856.

[35]

Penny, A. R. Changing the agenda for research into students' views about university teaching: Four shortcomings of srt research. Teaching in higher education 8, 3 (2003), 399--411.

[36]

Penny, A. R., and Coe, R. Effectiveness of consultation on student ratings feedback: A meta-analysis. Review of educational research 74, 2 (2004), 215--253.

[37]

Porter, S. R. Raising response rates: What works? New directions for institutional research 2004, 121 (2004), 5--21.

[38]

Pyasi, S., Gottipati, S., and Shankararaman, V. Sufat-an analytics tool for gaining insights from student feedback comments. In 2018 IEEE Frontiers in Education Conference (FIE) (2018), IEEE, pp. 1--9.

Digital Library

[39]

Richardson, J. T. Instruments for obtaining student feedback: A review of the literature. Assessment & evaluation in higher education 30, 4 (2005), 387--415.

[40]

Sax, L. J., Gilmartin, S. K., and Bryant, A. N. Assessing response rates and nonresponse bias in web and paper surveys. Research in higher education 44, 4 (2003), 409--432.

[41]

Schmelkin, L. P., Spencer, K. J., and Gellman, E. S. Faculty perspectives on course and teacher evaluations. Research in Higher Education 38 (1997), 575--592.

[42]

Simpson, R. D. Uses and misuses of student evaluations of teaching effectiveness. Innovative Higher Education 20, 1 (1995), 3--5.

[43]

Spencer, K. J., and Schmelkin, L. P. Student perspectives on teaching and its evaluation. Assessment & Evaluation in Higher Education 27, 5 (2002), 397--409.

[44]

Spooren, P., Brockx, B., and Mortelmans, D. On the validity of student evaluation of teaching: The state of the art. Review of Educational Research 83, 4 (2013), 598--642.

[45]

Svinicki, M. D. Encouraging your students to give feedback. New Directions for Teaching and Learning 2001, 87 (2001), 17--24.

[46]

Thorpe, S. W. Online student evaluation of instruction: An investigation of non-response bias. In 42nd Annual Forum of the Association for Institutional Research, Toronto, Ontario, Canada. ERIC Document No. ED472469 (2002).

[47]

Tourangeau, R. Remembering what happened: Memory errors and survey reports. In The science of self-report. Psychology Press, 1999, pp. 41--60.

[48]

Veeck, A., O'Reilly, K., MacMillan, A., and Yu, H. The use of collaborative midterm student evaluations to provide actionable results. Journal of Marketing Education 38, 3 (2016), 157--169.

[49]

Yao, Y., and Grady, M. L. How do faculty make formative use of student evaluation feedback?: A multiple case study. Journal of Personnel Evaluation in Education 18 (2005), 107--126.

[50]

Zabaleta, F. The use and misuse of student evaluations of teaching. Teaching in higher education 12, 1 (2007), 55--76.

Cited By

Yun JNie ABrunskill EDemszky D(2025)Exploring the Benefit of Customizing Feedback Interventions For Educators and Students With Offline Contextual Multi-Armed BanditsProceedings of the 15th International Learning Analytics and Knowledge Conference10.1145/3706468.3706551(944-949)Online publication date: 3-Mar-2025
https://dl.acm.org/doi/10.1145/3706468.3706551
Duncan ATang THuang YLuu JThakkar NJoyner DJoyner DKim MWang XXia M(2024)Forums, Feedback, and Two Kinds of AI: A Selective History of Learning @ ScaleProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664667(376-382)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3664667
Kumar HXiao RLawson BMusabirov IShi JWang XLuo HWilliams JRafferty AStamper JLiut MJoyner DKim MWang XXia M(2024)Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in ClassroomsProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3662042(86-97)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3662042

Index Terms

High-Resolution Course Feedback: Timely Feedback Mechanism for Instructors
1. Applied computing
  1. Education
    1. Computer-assisted instruction

Recommendations

Timely Feedback in Unstructured Cybersecurity Exercises
SIGCSE '18: Proceedings of the 49th ACM Technical Symposium on Computer Science Education

Cyber defence exercises are intensive, hands-on learning events for teams of professionals who gain or develop their skills to successfully prevent and respond to cyber attacks. The exercises mimic the real-life, routine operation of an organization ...
Examining the Factors that Influence how Instructors Provide Feedback in Online Learning Environments

Feedback is considered to be the bridge between what has been learned and what the student should know. Feedback can correct misconceptions, motivate learners, stimulate deep thinking, and guide future behavior. However, in the online environment ...
Effects of digital video-based feedback environments on pre-service teachers’ feedback competence
Abstract
The present study investigates the added value of blended digital video-based feedback environments in fostering pre-service teachers' feedback competence on teaching during a practicum. Pre-service teachers practised providing their ...
Highlights
- Effects of face-to-face and digital video-based feedback environments were compared.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

L@S '23: Proceedings of the Tenth ACM Conference on Learning @ Scale

July 2023

445 pages

ISBN:9798400700255

DOI:10.1145/3573051

General Chair:
Daniel Spikol
University of Copenhagen, Denmark
,
Program Chairs:
Olga Viberg
KTH Royal Institute of Technology, Sweden
,
Alejandra Martínez-Monés
Universidad de Valladolid, Spain
,
Philip Guo
UC San Diego, USA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

L@S '23

L@S '23: Tenth ACM Conference on Learning @ Scale

July 20 - 22, 2023

Copenhagen, Denmark

Acceptance Rates

Overall Acceptance Rate 117 of 440 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
202
Total Downloads

Downloads (Last 12 months)64
Downloads (Last 6 weeks)5

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yun JNie ABrunskill EDemszky D(2025)Exploring the Benefit of Customizing Feedback Interventions For Educators and Students With Offline Contextual Multi-Armed BanditsProceedings of the 15th International Learning Analytics and Knowledge Conference10.1145/3706468.3706551(944-949)Online publication date: 3-Mar-2025
https://dl.acm.org/doi/10.1145/3706468.3706551
Duncan ATang THuang YLuu JThakkar NJoyner DJoyner DKim MWang XXia M(2024)Forums, Feedback, and Two Kinds of AI: A Selective History of Learning @ ScaleProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664667(376-382)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3664667
Kumar HXiao RLawson BMusabirov IShi JWang XLuo HWilliams JRafferty AStamper JLiut MJoyner DKim MWang XXia M(2024)Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in ClassroomsProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3662042(86-97)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3662042

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten