Usability of Chat and Forum Discussion Tools in Higher Education
Authors: 
Tianyue Yang, Haoze Li, Md Athar Imtiaz, Eva Heinrich, Shannon Eijgenraam, Rachel BlagojevicAuthors Info & Claims
Pages 504 - 517
Abstract
Little is known about the usability and cognitive load experienced by students using online asynchronous discussion tools in higher education. Given that the use of both generic chat tools and specialised learning technology forum tools continue to increase, it is important for educators and designers/developers to understand the user experience of these tools. This research evaluates a text-based chat tool, Discord, and a text-based forum tool, Ed Discussion, from a usability and cognitive load perspective to complement the findings from the educational literature. We have identified usability issues that plague such tools related to learning curves, engagement, and effectiveness. Our findings can be used by educators to assist in choosing the right tools for their context and also make recommendations to designers of educational discussion tools.
References
[1]
Abbas, N., Online chat and chatbots to enhance mature student engagement in higher education. International Journal of Lifelong Education, 2022. 41(3): p. 308-326.
[2]
2. Tang, Y. and K.F. Hew, Effects of using mobile instant messaging on student behavioral, emotional, and cognitive engagement: a quasi-experimental study. International Journal of Educational Technology in Higher Education, 2022. 19(1): p. 3.
[3]
Lucho, S., G.B. Gardini, and J.M.E. Bueno. Teaching web development courses using flipped classroom and Discord: a two-year experience in the Peruvian context during the COVID-19 pandemic. in 2023 IEEE World Engineering Education Conference (EDUNINE). 2023. IEEE.
[4]
Santiago, A. and C. Mattos, From classroom education to remote emergency education: transformations in a dialogical pedagogy proposal:. Dialogic Pedagogy: An International Online Journal, 2023. 11(1): p. DT1-DT21.
[5]
Heinrich, E., H. Thomas, and E.R. Kahu, An exploration of course and cohort communication spaces in Discord, Teams, and Moodle. Australasian Journal of Educational Technology, 2022: p. 107-120.
[6]
Kahu, E.R., H.G. Thomas, and E. Heinrich, ‘A sense of community and camaraderie’: Increasing student engagement by supplementing an LMS with a Learning Commons Communication Tool. Active Learning in Higher Education, 2022: p. 146978742211276.
[7]
Arifianto, M. and I. Izzudin, Students’ acceptance of discord as an alternative online learning media. International Journal of Emerging Technologies in Learning (iJET), 2021. 16(20): p. 179-195.
[8]
Mora-Jimenez, L.D., User experience in communication and collaboration platforms: a comparative study including discord, microsoft teams, and zoom, in Information Technology and Systems: Proceedings of ICITS 2022. 2022, Springer. p. 52-61.
[9]
Uong, T.G.T., D.K. Nguyen, and H.N. Nguyen, Teachers’ Feedback on Using Discord as an Online Learning Platform. International Journal of TESOL & Education, 2022. 2(4): p. 84-104.
[10]
10. Ling, G., Learning with your Buddies: an investigation of community based UX design learning on Discord. 2022.
[11]
Tang, Y. and K.F. Hew, Examining the utility and usability of mobile instant messaging in a graduate-level course: A usefulness theoretical perspective. Australasian Journal of Educational Technology, 2019. 35(4).
[12]
Hameed, I., Social media usage and academic performance from a cognitive loading perspective. On the Horizon, 2022.
[13]
Van Nuland, S.E., R. Eagleson, and K.A. Rogers, Educational software usability: Artifact or Design? Anatomical sciences education, 2017. 10(2): p. 190-199.
[14]
Van Greunen, D. and J. Wesson, Formal usability testing of interactive educational software: A case study. Usability: Gaining a Competitive Edge, 2002: p. 161-176.
[15]
MacFarlane, S., G. Sim, and M. Horton. Assessing usability and fun in educational software. in Proceedings of the 2005 conference on Interaction design and children. 2005.
[16]
Costabile, M.F., On the usability evaluation of e-learning applications. in Proceedings of the 38th Annual Hawaii International Conference on System Sciences. 2005. IEEE.
[17]
Ardito, C., Usability of e-learning tools. in Proceedings of the working conference on Advanced visual interfaces. 2004.
[18]
Ardito, C., Towards guidelines for usability of e-learning applications. in User-Centered Interaction Paradigms for Universal Access in the Information Society: 8th ERCIM Workshop on User Interfaces for All, Vienna, Austria, June 28-29, 2004, Revised Selected Papers 8. 2004. Springer.
[19]
Gunesekera, A.I., Y. Bao, and M. Kibelloh, The role of usability on e-learning user interactions and satisfaction: A literature review. Journal of Systems and Information Technology, 2019.
[20]
Sweller, J., Cognitive load theory, in Psychology of learning and motivation. 2011, Elsevier. p. 37-76.
[21]
. Sweller, J., Cognitive Load Theory: Recent Theoretical Advances, in Cognitive Load Theory, J.L. Plass, R. Brünken, and R. Moreno, Editors. 2010, Cambridge University Press: Cambridge. p. 29-47.
[22]
Brignell, C., Fostering purposeful engagement by building staff-student communities. Student Engagement in Higher Education Journal, 2022. 4(2).
[23]
Ranga Auaha Ako (Learning and Teaching Design Team). Comparison - Piazza / Ed Discussion / Canvas Discussions. Available from: https://teachwell.auckland.ac.nz/resources/engagement/communication/#comparison.
[24]
. Bygstad, B., From dual digitalization to digital learning space: Exploring the digital transformation of higher education. Computers & Education, 2022. 182: p. 104463.
[25]
Loncar, M., N.E. Barrett, and G.-Z. Liu, Towards the refinement of forum and asynchronous online discussion in educational contexts worldwide: Trends and investigative approaches within a dominant research paradigm. Computers & Education, 2014. 73: p. 93-110.
[26]
Thomas, G. and S. Thorpe, Enhancing the facilitation of online groups in higher education: a review of the literature on face-to-face and online group-facilitation. Interactive Learning Environments, 2019. 27(1): p. 62-71.
[27]
Fehrman, S. and S.L. Watson, A Systematic Review of Asynchronous Online Discussions in Online Higher Education. American Journal of Distance Education, 2021. 35(3): p. 200-213.
[28]
De Lima, D.P.R., What to expect, and how to improve online discussion forums: the instructors’ perspective. Journal of Internet Services and Applications, 2019. 10(1): p. 22.
[29]
Griffin, L. and J. Roy, A great resource that should be utilised more, but also a place of anxiety: student perspectives on using an online discussion forum. Open Learning: The Journal of Open, Distance and e-Learning, 2022. 37(3): p. 235-250.
[30]
Manca, S., Snapping, pinning, liking or texting: Investigating social media in higher education beyond Facebook. The Internet and Higher Education, 2020. 44: p. 100707.
[31]
Vladoiu, M. and Z. Constantinescu. Learning During COVID-19 Pandemic: Online Education Community, Based on Discord. in 2020 19th RoEduNet Conference: Networking in Education and Research (RoEduNet). 2020. IEEE.
[32]
Hernández-Lara, A.B., A. Perera-Lluna, and E. Serradell-López, Game learning analytics of instant messaging and online discussion forums in higher education. Education + Training, 2021. 63(9): p. 1288-1308.
[33]
Squires, D. and J. Preece, Usability and learning: evaluating the potential of educational software. Computers & education, 1996. 27(1): p. 15-22.
[34]
Chu, A., Usability of Learning Moment: Features of an E-learning Tool That Maximize Adoption by Students. West J Emerg Med, 2019. 21(1): p. 78-84.
[35]
Kakasevski, G., Evaluating usability in learning management system moodle. in ITI 2008 - 30th International Conference on Information Technology Interfaces. 2008.
[36]
Althobaiti, M.M. and P. Mayhew. Assessing the Usability of Learning Management System: User Experience Study. in E-Learning, E-Education, and Online Training. 2016. Cham: Springer International Publishing.
[37]
Abuhlfaia, K. and E.d. Quincey, The usability of E-learning platforms in higher education: a systematic mapping study, in Proceedings of the 32nd International BCS Human Computer Interaction Conference. 2018, BCS Learning & Development Ltd.: Belfast, United Kingdom. p. Article 7.
[38]
Abuhlfaia, K. and E. De Quincey, The Usability of E-learning Platforms in Higher Education: A Systematic Mapping Study. 2018. 1-13.
[39]
Masood, M. and A. Musman, The Usability and its Influence of an e-Learning System on Student Participation. Procedia - Social and Behavioral Sciences, 2015. 197: p. 2325-2330.
[40]
Kubacz-Szumska, J. and O. Szumski, Cloud Communications During the Pandemic From the Perspective of Collaboration Platforms. Problemy Zarządzania, 2021. 19(3/2021 (93): p. 138-149.
[41]
Cuijpers, J., An empirical evaluation of video conferencing systems used in industry, academia, and entertainment. in Companion of the ACM/SPEC International Conference on Performance Engineering. 2021.
[42]
Sidpra, J., Sustaining education in the age of COVID-19: a survey of synchronous web-based platforms. Quantitative Imaging in Medicine and Surgery, 2020. 10(7): p. 1422.
[43]
Keller, S., J. Pirker, and E. List. Digital Communication Tools in Private and Professional Environments. in International Conference on Interactive Collaborative Learning. 2022. Springer.
[44]
Dayana, Y.E., O.M. Andre, and L. Andrade-Arenas. Design of the Discord application as an E-learning tool at the University of Sciences and Humanities. in LACCEI International Multi-Conference for Engineering, Education, and Technology. 2020.
[45]
Shi, Z., G. Luo, and L. He, Mobile-assisted Language Learning Using WeChat Instant Messaging. International Journal of Emerging Technologies in Learning (iJET), 2017. 12(02): p. pp. 16-26.
[46]
Young, M.S., State of science: mental workload in ergonomics. Ergonomics, 2015. 58(1): p. 1-17.
[47]
Paas, F.G., Training strategies for attaining transfer of problem-solving skill in statistics: a cognitive-load approach. Journal of educational psychology, 1992. 84(4): p. 429.
[48]
Hart, S.G. and L.E. Staveland, Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research, in Advances in Psychology, P.A. Hancock and N. Meshkati, Editors. 1988, North-Holland. p. 139-183.
[49]
Marquart, G., C. Cabrall, and J. de Winter, Review of Eye-related Measures of Drivers’ Mental Workload. Procedia Manufacturing, 2015. 3: p. 2854-2861.
[50]
Bagley, J. and L. Manelis, Effect of awareness on an indicator of cognitive load. Perceptual and Motor Skills, 1979. 49(2): p. 591-594.
[51]
Korbach, A., R. Brünken, and B. Park, Measurement of cognitive load in multimedia learning: a comparison of different objective measures. Instructional science, 2017. 45: p. 515-536.
[52]
Brooke, J., SUS-A quick and dirty usability scale. Usability evaluation in industry, 1996. 189(194): p. 4-7.
[53]
Lewis, C. and J. Rieman, Task-centered user interface design. A practical introduction, 1993.
[54]
Tonsen, M., C.K. Baumann, and K. Dierkes, A high-level description and performance evaluation of pupil invisible. arXiv preprint arXiv:2009.00508, 2020.
[55]
Klingner, J., Measuring cognitive load during visual tasks by combining pupillometry and eye tracking. 2010: Stanford University.
[56]
Bangor, A., P. Kortum, and J. Miller, Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies, 2009. 4(3): p. 114-123.
[57]
Fuhl, W., Pupil detection for head-mounted eye tracking in the wild: an evaluation of the state of the art. Machine Vision and Applications, 2016. 27: p. 1275-1288.
[58]
58. Goldberg, J.H. and A.M. Wichansky, Eye tracking in usability evaluation: A practitioner's guide, in the Mind's Eye. 2003, Elsevier. p. 493-516.
[59]
Poole, A., Ball. LJ (2005). Eye tracking in human-computer interaction and usability research: current status and future prospects. Encyclopedia of Human-Computer Interaction: p. 211-219.
[60]
Kwak, S.K. and J.H. Kim, Statistical data preparation: management of missing values and outliers. Korean journal of anesthesiology, 2017. 70(4): p. 407-411.
[61]
. McHugh, M.L., Interrater reliability: the kappa statistic. Biochem Med (Zagreb), 2012. 22(3): p. 276-82.
[62]
McCrum-Gardner, E., Which is the correct statistical test to use? British Journal of Oral and Maxillofacial Surgery, 2008. 46(1): p. 38-41.
[63]
. Stephen, D. and A. SAZ, Cochran's Q with pairwise McNemar for dichotomous multiple responses data: A practical approach. Int J Eng Technol, 2018. 7(3): p. 4-6.
[64]
Watson, A.B. and J.I. Yellott, A unified formula for light-adapted pupil size. Journal of vision, 2012. 12(10): p. 12-12.
[65]
Nielsen, J. and T.K. Landauer, A mathematical model of the finding of usability problems, in Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems. 1993, Association for Computing Machinery: Amsterdam, The Netherlands. p. 206–213.
[66]
Dumas, J.S. and J.C. Redish, A Practical Guide to Usability Testing. 1999: Intellect Books.
Index Terms
- Usability of Chat and Forum Discussion Tools in Higher Education
Recommendations
Student and Instructor Satisfaction with E-learning Tools in Online Learning Environments
This study utilized Expectancy Confirmation Theory ECT and Technology Acceptance Model TAM to examine satisfaction of students and instructors toward online learning tools and resources in online learning environments. The study findings indicate that ...
Online formative assessment in higher education: A review of the literature
As online and blended learning has become common place educational strategy in higher education, educators need to reconceptualise fundamental issues of teaching, learning and assessment in non traditional spaces. These issues include concepts such as ...
Effects of Competitive E-Learning Tools on Higher Education Students: A Case Study
Over the last few years, most of the attempts to introduce active learning methodologies in the classroom have made use of Information and Communication Technology (ICT). Many of these efforts have been directed to collaborative scenarios used in remote,...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2023
733 pages
ISBN:9798400717079
DOI:10.1145/3638380
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 10 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
OzCHI 2023
Acceptance Rates
Overall Acceptance Rate 362 of 729 submissions, 50%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 46Total Downloads
- Downloads (Last 12 months)46
- Downloads (Last 6 weeks)10
Reflects downloads up to 12 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format