research-article

Open access

Post Recommendation System Impact on Student Participation and Performance in an Online AI Graduate Course

Authors:

Saurabh Chatterjee,

Roy Finkelberg,

Thad StarnerAuthors Info & Claims

L@S '22: Proceedings of the Ninth ACM Conference on Learning @ Scale

Pages 24 - 34

https://doi.org/10.1145/3491140.3528280

Published: 01 June 2022 Publication History

Abstract

Embedding a post recommendation system in online course forums improves students' ability to find relevant course content. Yet, there is limited research quantifying how these interventions impact forum interactions and students' class performance. We randomly divide an on-line class in a Masters degree-seeking program into control and experimental sections. Midway through the class, for the experimental section, we introduce an information retrieval system that suggests relevant posts to students while they compose their new posts. The average number of initial posts in the discussion forum dropped by 55% for the experimental group compared to 21% in the control group. In post-hoc analysis, we classify students as having an active or passive (i.e., mostly lurking) forum participation style. Posts per student per assignment by passive participants in the experimental group decreased by 15%, while passive participants' posts in the control group increased by 200%. The number of answers given by instructors in the experimental group decreased twice as much as the control group after intervention, though the difference was not statistically significant. The reduction of posts was not associated with a decrease in academic performance (grades) for the experimental group. This line of research convinced the popular forum Piazza to implement a post recommendation system similar to that used in these experiments.

References

[1]

Akshay Agrawal, Jagadish Venkatraman, Shane Leonard, and Andreas Paepcke. 2015. YouEDU: addressing confusion in MOOC discussion forums by recommending instructional video clips. (2015).

[2]

Ma Almeda et al . 2018. Comparing the Factors That Predict Completion and Grades Among For-Credit and Open/MOOC Students in Online Learning. Online Learning 22, 1 (2018), 1--18.

[3]

Ya-Hui An, Muthu Kumar Chandresekaran, Min-Yen Kan, and Yan Fu. 2018. The MUIR Framework: Cross-linking MOOC resources to enhance discussion forums. In International Conference on Theory and Practice of Digital Libraries. Springer, 208--219.

[4]

Fernanda Bonafini, Chungil Chae, Eunsung Park, and Kathryn Jablokow. 2017. How Much Does Student Engagement with Videos and Forums in a MOOC Affect Their Achievement? Online Learning Journal 21, 4 (December 2017). https://www.learntechlib.org/p/183772

[5]

Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Céspedes, Steve Yuan, Chris Tar, et al . 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175 (2018).

[6]

Fei-ching Chen. 2004. Passive forum behaviors (lurking): A community perspective. In Embracing Diversity in the Learning Sciences: Proceedings of the Sixth International Conference of the Learning Sciences. Psychology Press, 128.

[7]

Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. 2018. Quora question pairs. URL https://www. kaggle.com/c/quora-question-pairs (2018).

[8]

Cho Kin Cheng, Dwayne E Paré, Lisa-Marie Collimore, and Steve Joordens. 2011. Assessing the effectiveness of a voluntary online discussion forum on improving students' course performance. Computers & Education 56, 1 (2011), 253--261.

Digital Library

[9]

Mary M Christopher, Julie A Thomas, and Mary K Tallent-Runnels. 2004. Raising the bar: Encouraging high level thinking in online discussion forums. Roeper Review 26, 3 (2004), 166--171.

[10]

Derrick Coetzee, Armando Fox, Marti A Hearst, and Björn Hartmann. 2014. Should your MOOC forum use a reputation system?. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 1176--1187.

Digital Library

[11]

Aubteen Darabi, Meagan C Arrastia, David W Nelson, Tom Cornille, and Xinya Liang. 2011. Cognitive presence in asynchronous online learning: A comparison of four discussion strategies. Journal of Computer Assisted Learning 27, 3 (2011), 216--227.

[12]

Jo Davies and Martin Graff. 2005. Performance in e-learning: online participation and student grades. British Journal of Educational Technology 36, 4 (2005), 657--663.

[13]

Paula G de Barba, Gregor E Kennedy, and Mary D Ainley. 2016. The role of students' motivation and participation in predicting performance in a MOOC. Journal of Computer Assisted Learning 32, 3 (2016), 218--231.

Digital Library

[14]

Franz Faul, Edgar Erdfelder, Albert-Georg Lang, and Axel Buchner. 2007. G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior research methods 39, 2 (2007), 175--191.

[15]

Rebecca Ferguson and Doug Clow. 2015. Examining engagement: analysing learner subpopulations in massive open online courses (MOOCs). In Proceedings of the fifth international conference on learning analytics and knowledge. 51--58.

Digital Library

[16]

D Randy Garrison and J Ben Arbaugh. 2007. Researching the community of inquiry framework: Review, issues, and future directions. The Internet and higher education 10, 3 (2007), 157--172.

[17]

India Irish, Roy Finkelberg, Daniel Nkemelu, Swar Gujrania, Aadarsh Padiyath, Sumedha Raman, Chirag Tailor, Rosa Arriaga, and Thad Starner. 2020. PARQR: Automatic Post Suggestion in the Piazza Online Forum to Support Degree Seeking Online Masters Students. In Proceedings of the Seventh ACM Conference on Learning@ Scale. 125--134.

Digital Library

[18]

Laura Jacobi. 2017. The structure of discussions in an online communication Course: What do students find most effective? Journal of University Teaching & Learning Practice 14, 1 (2017), 11.

[19]

Thorsten Joachims. 1996. A Probabilistic Analysis of the Rocchio Algorithm with TFIDF for Text Categorization. Technical Report. Carnegie-mellon univ pittsburgh pa dept of computer science.

[20]

Quentin Jones, Gilad Ravid, and Sheizaf Rafaeli. 2004. Information overload and the message dynamics of online interaction spaces: A theoretical model and empirical exploration. Information systems research 15, 2 (2004), 194--210.

[21]

Tali Kahan, Tal Soffer, and Rafi Nachmias. 2017. Types of participant behavior in a massive open online course. International Review of Research in Open and Distributed Learning: IRRODL 18, 6 (2017), 1--18.

[22]

A Kardan, Amir Narimani, and Foozhan Ataiefard. 2017. A hybrid approach for thread recommendation in MOOC forums. International Journal of Social, Behavioral, Educational, Economic, Business and Industrial Engineering 11, 10 (2017), 2195--2201.

[23]

Jungjoo Kim. 2013. Influence of group size on students' participation in online discussion forums. Computers & Education 62 (2013), 123--129.

[24]

René F Kizilcec, Chris Piech, and Emily Schneider. 2013. Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. In Proceedings of the third international conference on learning analytics and knowledge. 170--179.

Digital Library

[25]

Cath Mazuro and Namrata Rao. 2011. Online discussion forums in higher education: Is lurking working. International Journal for Cross-Disciplinary Subjects in Education 2, 2 (2011), 364--371.

[26]

Joanne M McInnerney and Tim S Roberts. 2004. Online learning: Social interaction and the creation of a sense of community. Journal of Educational Technology & Society 7, 3 (2004), 73--81.

[27]

Nicolas Michinov, Sophie Brunot, Olivier Le Bohec, Jacques Juhel, and Marine Delaval. 2011. Procrastination, participation, and performance in online learning environments. Computers & Education 56, 1 (2011), 243--252.

Digital Library

[28]

Eni Mustafaraj and Jessica Bu. 2015. The visible and invisible in a MOOC discussion forum. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale. 351--354.

Digital Library

[29]

Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532--1543.

[30]

Reihaneh Rabbany k, Mansoureh Takaffoli, and Osmar R Zaiane. 2012. Social network analysis and mining to support the assessment of on-line student participation. ACM SIGKDD Explorations Newsletter 13, 2 (2012), 20--29.

Digital Library

[31]

Radim Rehurek, Petr Sojka, et al . 2011. Gensim-statistical semantics in python. Retrieved from genism. org (2011).

[32]

Bradley Rhodes and Thad Starner. 1996. Remembrance Agent: A continuously running automated information retrieval system. In The Proceedings of The First International Conference on The Practical Application Of Intelligent Agents and Multi Agent Technology. 487--495.

[33]

Barbara Rogoff. 1994. Developing understanding of the idea of communities of learners. Mind, culture, and activity 1, 4 (1994), 209--229.

[34]

Cristóbal Romero, Manuel-Ignacio López, Jose-María Luna, and Sebastián Ventura. 2013. Predicting students' final performance from participation in on-line discussion forums. Computers & Education 68 (2013), 458--472.

[35]

Derek Rowntree. 1995. Teaching and learning online: a correspondence education for the 21st century? British journal of educational technology 26, 3 (1995), 205--215.

[36]

Nicholas P. Salter and Marissa R. Conneely. 2015. Structured and unstructured discussion forums as tools for student engagement. Computers in Human Behavior 46 (2015), 18--25. https://doi.org/10.1016/j.chb.2014.12.037

Digital Library

[37]

Sarah Schrire. 2006. Knowledge building in asynchronous discussion groups: Going beyond quantitative analysis. Computers & Education 46, 1 (2006), 49--70.

Digital Library

[38]

Afsaneh Sharif and Barry Magrill. 2015. Discussion forums in MOOCs. International Journal of Learning, Teaching and Educational Research 12, 1 (2015).

[39]

Donna Smith and Katy Smith. 2014. The Case for 'Passive' Learning--The 'Silent' Community of Online Learners. European Journal of Open, Distance and e-learning 17, 2 (2014).

[40]

T. Starner. 1994. The Cyborgs Are Coming. Technical Report 318. Perceptual Computing, MIT Media Laboratory.

[41]

Selma Vonderwell and Sajit Zachariah. 2005. Factors that influence participation in online learning. Journal of Research on Technology in education 38, 2 (2005), 213--230.

[42]

Alyssa Friend Wise, Yi Cui, WanQi Jin, and Jovita Vytasek. 2017. Mining for gold: Identifying content-related MOOC discussion threads across domains through linguistic modeling. The Internet and Higher Education 32 (2017), 11--28.

[43]

Alyssa Friend Wise, Yi Cui, and Jovita Vytasek. 2016. Bringing order to chaos in MOOC discussion forums with content-related thread identification. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge. 188--197.

Digital Library

[44]

Dezhi Wu and Starr Roxanne Hiltz. 2003. Online discussions and perceived learning. AMCIS 2003 Proceedings (2003), 86.

[45]

Diyi Yang, David Adamson, and Carolyn Penstein Rosé. 2014. Question recommendation with constraints for massive open online courses. In Proceedings of the 8th ACM Conference on Recommender systems. 49--56.

Digital Library

[46]

Diyi Yang, Jingbo Shang, and Carolyn Penstein Rosé. 2014. Constrained question recommendation in MOOCs via submodularity. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. 1987--1990.

Digital Library

[47]

Brian Zylich, Adam Viola, Brokk Toggerson, Lara Al-Hariri, and Andrew Lan. 2020. Exploring automated question answering methods for teaching assistance. In International Conference on Artificial Intelligence in Education. Springer, 610--622.

Digital Library

Cited By

Duncan ATang THuang YLuu JThakkar NJoyner DJoyner DKim MWang XXia M(2024)Forums, Feedback, and Two Kinds of AI: A Selective History of Learning @ ScaleProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664667(376-382)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3664667
Plaza LAraujo LLópez-Ostenero FMartínez-Romo J(2023)Automatic Recommendation of Forum Threads and Reinforcement Activities in a Data Structure and Programming CourseApplied System Innovation10.3390/asi60500836:5(83)Online publication date: 21-Sep-2023
https://doi.org/10.3390/asi6050083
Irish IChatterjee SJivani SJia XLee JArriaga RStarner TSpikol DViberg OMartínez-Monés AGuo P(2023)Managing the Chaos: Approaches to Navigating Discussion Forums for Instructional StaffProceedings of the Tenth ACM Conference on Learning @ Scale10.1145/3573051.3596197(406-410)Online publication date: 20-Jul-2023
https://dl.acm.org/doi/10.1145/3573051.3596197
Show More Cited By

Index Terms

Post Recommendation System Impact on Student Participation and Performance in an Online AI Graduate Course
1. Applied computing
  1. Education
2. Information systems
  1. Information retrieval

Index terms have been assigned to the content through auto-classification.

Recommendations

Prompting authentic blogging practice in an online graduate course

Blogging is characterized by an individual exploration of ideas of personal interest through frequent online posts, documenting ideas as they emerge over time. Community emerges as bloggers read and link across blogs, based on shared interests. Blogs ...
Non-public and public online community participation: Needs, attitudes and behavior

Nonpublic participation within an online community, often called lurking, occurs when an individual joins a community, but does not post. This study examines the nature of lurking, why people lurk and the differences in attitudes between lurkers and ...
A system for extracting and comparing memes in online forums

Automatic extraction of top memes (key concepts, relations) from unstructured text.Sophisticated meme construction and matching algorithm.System with default/customizable control parameter set.No preprocessing of text or manual intervention between ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

L@S '22: Proceedings of the Ninth ACM Conference on Learning @ Scale

June 2022

491 pages

ISBN:9781450391580

DOI:10.1145/3491140

General Chair:
René F. Kizilcec
Cornell University, USA
,
Program Chairs:
Katie Davis
University of Washington, USA
,
Xavier Ochoa
New York University, USA

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 June 2022

Check for updates

Author Tags

Qualifiers

Research-article

Conference

L@S '22

L@S '22: Ninth (2022) ACM Conference on Learning @ Scale

June 1 - 3, 2022

NY, New York City, USA

Acceptance Rates

Overall Acceptance Rate 117 of 440 submissions, 27%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
2,025
Total Downloads

Downloads (Last 12 months)1,688
Downloads (Last 6 weeks)719

Reflects downloads up to 14 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Duncan ATang THuang YLuu JThakkar NJoyner DJoyner DKim MWang XXia M(2024)Forums, Feedback, and Two Kinds of AI: A Selective History of Learning @ ScaleProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664667(376-382)Online publication date: 9-Jul-2024
https://dl.acm.org/doi/10.1145/3657604.3664667
Plaza LAraujo LLópez-Ostenero FMartínez-Romo J(2023)Automatic Recommendation of Forum Threads and Reinforcement Activities in a Data Structure and Programming CourseApplied System Innovation10.3390/asi60500836:5(83)Online publication date: 21-Sep-2023
https://doi.org/10.3390/asi6050083
Irish IChatterjee SJivani SJia XLee JArriaga RStarner TSpikol DViberg OMartínez-Monés AGuo P(2023)Managing the Chaos: Approaches to Navigating Discussion Forums for Instructional StaffProceedings of the Tenth ACM Conference on Learning @ Scale10.1145/3573051.3596197(406-410)Online publication date: 20-Jul-2023
https://dl.acm.org/doi/10.1145/3573051.3596197
Murad DToha MMayatopani HWijanarko BHeryadi YDewi MLeandros R(2023)Personalized Recommendation System for Online Learning: An Opportunity2023 8th International Conference on Business and Industrial Research (ICBIR)10.1109/ICBIR57571.2023.10147613(128-132)Online publication date: 18-May-2023
https://doi.org/10.1109/ICBIR57571.2023.10147613

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten