Article

New Measures for Offline Evaluation of Learning Path Recommenders

Authors:

Anne BoyerAuthors Info & Claims

Addressing Global Challenges and Quality Education: 15th European Conference on Technology Enhanced Learning, EC-TEL 2020, Heidelberg, Germany, September 14–18, 2020, Proceedings

Pages 259 - 273

https://doi.org/10.1007/978-3-030-57717-9_19

Published: 14 September 2020 Publication History

Abstract

Recommending students useful and effective learning paths is highly valuable to improve their learning experience. The evaluation of the effectiveness of this recommendation is a challenging task that can be performed online or offline. Online evaluation is highly popular but it relies on actual path recommendations to students, which may have dramatic implications. Offline evaluation relies on static datasets of students’ learning activities and simulates paths recommendations. Although easier to run, it is difficult to accurately evaluate offline the effectiveness of a learning path recommendation. To tackle this issue, this work proposes simple offline evaluation measures. We show that they actually allow to characterise and differentiate the algorithms.

References

[1]

Durand, G., Laplante, F., Kop, R.: A learning design recommendation system based on Markov decision processes. In: 17th ACM KDD (2011)

[2]

Durand G, Belacel N, and LaPlante F Graph theory based model for learning path recommendation Inf. Sci. 2013 251 10-21

[3]

Dwivedi P, Kant V, and Bharadwaj K Learning path recommendation based on modified variable length genetic algorithm Educ. Inf. Technol. 2018 23 2 819-836

[4]

Embretson SE and Reise SP Item Response Theory 2013 New York Psychology Press

[5]

Feng, S., Li, X., Zeng, Y., Cong, G., Chee, Y.M., Yuan, Q.: Personalized ranking metric embedding for next new POI recommendation. In: 24th IJCAI (2015)

[6]

Hsieh T and Wang T A mining-based approach on discovering courses pattern for constructing suitable learning path Expert Syst. Appl. 2010 37 6 4156-4167

[7]

Léonard M, Peter Y, and Secq Y Scheffel M, Broisin J, Pammer-Schindler V, Ioannou A, and Schneider J Patterns and loops: early computational thinking Transforming Learning with Meaningful Technologies 2019 Cham Springer 280-293

[8]

Liu, Q., et al.: Exploiting cognitive structure for adaptive learning. In: Proceedings of the 25th KDD, pp. 627–635 (2019)

[9]

Monti D, Palumbo E, Rizzo G, and Morisio M Sequeval: an offline evaluation framework for sequence-based RS Information 2019 10 5 174

[10]

Nabizadeh, A., Gonçalves, D., Gama, S., Jorge, J., Rafsanjani, H.: Adaptive learning path recommender approach using auxiliary learning objects. Comput. Educ. 147 (2020)

[11]

Nabizadeh A, Jorge A, and Leal J Estimating time and score uncertainty in generating successful learning paths under time constraints Expert Syst. 2019 36 2 e12351

[12]

Nabizadeh, A., Jorge, A., Leal, J.P.: Long term goal oriented recommender systems. In: Proceedings of the 11th WEBIST, pp. 552–557 (2015)

[13]

Quadrana M, Cremonesi P, and Jannach D Sequence-aware recommender systems ACM Comput. Surv. (CSUR) 2018 51 4 1-36

[14]

Rossetti, M., Stella, F., Zanker, M.: Contrasting offline and online results when evaluating recommendation algorithms. In: Proceedings of the 10th RecSys, pp. 31–34 (2016)

[15]

Su C Designing and developing a novel hybrid adaptive learning path recommendation system (ALPRS) for gamification mathematics geometry course Eurasia J. Math. Sci. Technol. Educ. 2017 13 6 2275-2298

[16]

Su J, Tseng S, Wang W, Weng J, Yang J, and Tsai W Learning portfolio analysis and mining for SCORM compliant environment J. Educ. Technol. Soc. 2006 9 1 262-275

[17]

Taraghi B, Saranti A, Ebner M, and Schön M Zaphiris P and Ioannou A Markov chain and classification of difficulty levels enhances the learning path in one digit multiplication International Conference on Learning and Collaboration Technologies. Designing and Developing Novel Learning Experiences 2014 Cham Springer 322-333

[18]

Venant R, Sharma K, Vidal P, Dillenbourg P, and Broisin J Lavoué É, Drachsler H, Verbert K, Broisin J, and Pérez-Sanagustín M Using sequential pattern mining to explore learners’ behaviors and evaluate their correlation with performance in inquiry-based learning Data Driven Approaches in Digital Education 2017 Cham Springer 286-299

[19]

Vesin B, Klašnja-Milićević A, Ivanović M, and Budimac Z Applying recommender systems and adaptive hypermedia for e-learning personalization Comput. Inform. 2013 32 3 629-659

[20]

Xie H, Zou D, Wang FL, Wong TL, Rao Y, and Wang SH Discover learning path for group users: a profile-based approach Neurocomputing 2017 254 59-70

[21]

Zhou Y, Huang C, Hu Q, Zhu J, and Tang Y Personalized learning full-path recommendation model based on LSTM neural networks Inf. Sci. 2018 444 135-152

[22]

Zhu H et al. A multi-constraint learning path recommendation algorithm based on knowledge map Knowl.-Based Syst. 2018 143 102-114

Cited By

Zhang ZBrun ABoyer A(2022)Managing Learners’ Memory Strength in a POMDP-Based Learning Path Recommender SystemArtificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners’ and Doctoral Consortium10.1007/978-3-031-11647-6_53(284-288)Online publication date: 27-Jul-2022
https://dl.acm.org/doi/10.1007/978-3-031-11647-6_53

Recommendations

A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation
RepSys '13: Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation

Offline evaluations are the most common evaluation method for research paper recommender systems. However, no thorough discussion on the appropriateness of offline evaluations has taken place, despite some voiced criticism. We conducted a study in which ...
Bridging the Gap Between User-centric and Offline Evaluation of Personalized Recommendation Systems
UMAP '18: Adjunct Publication of the 26th Conference on User Modeling, Adaptation and Personalization

In this paper, we propose to evaluate recommender systems by conducting both offline and user-centric evaluations, while considering multiple quality aspects in realistic settings. This comprehensive evaluation would provide insight on how to improve ...
Estimating Error and Bias in Offline Evaluation Results
CHIIR '20: Proceedings of the 2020 Conference on Human Information Interaction and Retrieval

Offline evaluations of recommender systems attempt to estimate users' satisfaction with recommendations using static data from prior user interactions. These evaluations provide researchers and developers with first approximations of the likely ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

Addressing Global Challenges and Quality Education: 15th European Conference on Technology Enhanced Learning, EC-TEL 2020, Heidelberg, Germany, September 14–18, 2020, Proceedings

Sep 2020

507 pages

ISBN:978-3-030-57716-2

DOI:10.1007/978-3-030-57717-9

Editors:
Carlos Alario-Hoyos
Universidad Carlos III de Madrid, Leganés (Madrid), Spain
,
María Jesús Rodríguez-Triana
Tallinn University, Tallinn, Estonia
,
Maren Scheffel
Open University Netherlands, Heerlen, The Netherlands
,
Inmaculada Arnedillo-Sánchez
Trinity College Dublin, Dublin, Ireland
,
Sebastian Maximilian Dennerlein
Graz University of Technology and Know-Center, Graz, Austria

© Springer Nature Switzerland AG 2020.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 14 September 2020

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang ZBrun ABoyer A(2022)Managing Learners’ Memory Strength in a POMDP-Based Learning Path Recommender SystemArtificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners’ and Doctoral Consortium10.1007/978-3-031-11647-6_53(284-288)Online publication date: 27-Jul-2022
https://dl.acm.org/doi/10.1007/978-3-031-11647-6_53

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents