Nothing Special   »   [go: up one dir, main page]

skip to main content
10.1145/3437963.3441747acmconferencesArticle/Chapter ViewAbstractPublication PageswsdmConference Proceedingsconference-collections
research-article

Federated Deep Knowledge Tracing

Published: 08 March 2021 Publication History

Abstract

Knowledge tracing is a fundamental task in intelligent education for tracking the knowledge states of students on necessary concepts. In recent years, Deep Knowledge Tracing (DKT) utilizes recurrent neural networks to model student learning sequences. This approach has achieved significant success and has been widely used in many educational applications. However, in practical scenarios, it tends to suffer from the following critical problems due to data isolation: 1) Data scarcity. Educational data, which is usually distributed across different silos (e.g., schools), is difficult to gather. 2) Different data quality. Students in different silos have different learning schedules, which results in unbalanced learning records, meaning that it is necessary to evaluate the learning data quality independently for different silos. 3) Data incomparability. It is difficult to compare the knowledge states of students with different learning processes from different silos. Inspired by federated learning, in this paper, we propose a novel Federated Deep Knowledge Tracing (FDKT) framework to collectively train high-quality DKT models for multiple silos. In this framework, each client takes charge of training a distributed DKT model and evaluating data quality by leveraging its own local data, while a center server is responsible for aggregating models and updating the parameters for all the clients. In particular, in the client part, we evaluate data quality incorporating different education measurement theories, and we construct two quality-oriented implementations based on FDKT, i.e., FDKTCTT and FDKTIRT-where the means of data quality evaluation follow Classical Test Theory and Item Response Theory, respectively. Moreover, in the server part, we adopt hierarchical model interpolation to uptake local effects for model personalization. Extensive experiments on real-world datasets demonstrate the effectiveness and superiority of the FDKT framework.

References

[1]
Ghodai Abdelrahman and Qing Wang. 2019. Knowledge Tracing with Sequential Key-Value Memory Networks. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 175--184.
[2]
Nabeel Abedalaziz and Chin Hai Leng. 2018. The relationship between CTT and IRT approaches in Analyzing Item Characteristics. MOJES: Malaysian Online Journal of Educational Sciences, Vol. 1, 1 (2018), 64--70.
[3]
Allan Birnbaum. 1969. Statistical theory for logistic mental test models with a prior distribution of ability. Journal of Mathematical Psychology, Vol. 6, 2 (1969), 258--276.
[4]
Léon Bottou. 2010. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010. Springer, 177--186.
[5]
Yuying Chen, Qi Liu, Zhenya Huang, Le Wu, Enhong Chen, Runze Wu, Yu Su, and Guoping Hu. 2017. Tracking knowledge proficiency of students with educational priors. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 989--998.
[6]
Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, Vol. 4, 4 (1994), 253--278.
[7]
Lee J Cronbach. 1951. Coefficient alpha and the internal structure of tests. psychometrika, Vol. 16, 3 (1951), 297--334.
[8]
Barbara B Ellis and Alan D Mead. 2002. Item analysis: Theory and practice using classical and modern test theory. (2002).
[9]
Harold Gulliksen. 1950. Theory of mental tests. (1950).
[10]
Nina Guyon, Eric Maurin, and Sandra McNally. 2012. The effect of tracking students by ability into different schools a natural experiment. Journal of Human Resources, Vol. 47, 3 (2012), 684--721.
[11]
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Francc oise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. 2018. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604 (2018).
[12]
Li Huang, Yifeng Yin, Zeng Fu, Shifa Zhang, Hao Deng, and Dianbo Liu. 2018. LoAdaBoost: Loss-Based AdaBoost Federated Machine Learning on medical Data. arXiv preprint arXiv:1811.12629 (2018).
[13]
Zhenya Huang, Qi Liu, Enhong Chen, Hongke Zhao, Mingyong Gao, Si Wei, Yu Su, and Guoping Hu. 2017. Question Difficulty Prediction for READING Problems in Standard Tests. In AAAI. 1352--1359.
[14]
Zhenya Huang, Qi Liu, Yuying Chen, Le Wu, Keli Xiao, Enhong Chen, Haiping Ma, and Guoping Hu. 2020. Learning or Forgetting? A Dynamic Approach for Tracking the Knowledge Proficiency of Students. ACM Transactions on Information Systems (TOIS), Vol. 38, 2 (2020), 1--33.
[15]
Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Seong-Lyun Kim. 2018. Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data. arXiv preprint arXiv:1811.11479 (2018).
[16]
Shaoxiong Ji, Shirui Pan, Guodong Long, Xue Li, Jing Jiang, and Zi Huang. 2019. Learning private neural language modeling with attentive aggregation. In 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 1--8.
[17]
Di Jiang, Yuanfeng Song, Yongxin Tong, Xueyang Wu, Weiwei Zhao, Qian Xu, and Qiang Yang. 2019. Federated Topic Modeling. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 1071--1080.
[18]
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. 2019. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 (2019).
[19]
Eugene Kharitonov. 2019. Federated online learning to rank with evolution strategies. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. 249--257.
[20]
G Gage Kingsbury and David J Weiss. 1983. A comparison of IRT-based adaptive mastery testing and a sequential mastery testing procedure. In New horizons in testing. Elsevier, 257--283.
[21]
George D Kuh, Jillian Kinzie, Jennifer A Buckley, Brian K Bridges, and John C Hayek. 2011. Piecing together the student success puzzle: research, propositions, and recommendations: ASHE Higher Education Report. Vol. 116. John Wiley & Sons.
[22]
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2018. Federated optimization in heterogeneous networks. arXiv preprint arXiv:1812.06127 (2018).
[23]
Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. 2019. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189 (2019).
[24]
Qi Liu, Enhong Chen, Hui Xiong, Chris HQ Ding, and Jian Chen. 2011a. Enhancing collaborative filtering by user interest expansion via personalized ranking. T SYST MAN, Vol. 42, 1 (2011), 218--233.
[25]
Qi Liu, Yong Ge, Zhongmou Li, Enhong Chen, and Hui Xiong. 2011b. Personalized travel package recommendation. In 2011 IEEE 11th International Conference on Data Mining. IEEE, 407--416.
[26]
Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. 2019. Ekt: Exercise-aware knowledge tracing for student performance prediction. IEEE Transactions on Knowledge and Data Engineering, Vol. 33, 1 (2019), 100--115.
[27]
Qi Liu, Runze Wu, Enhong Chen, Guandong Xu, Yu Su, Zhigang Chen, and Guoping Hu. 2018. Fuzzy cognitive diagnosis for modelling examinee performance. ACM Transactions on Intelligent Systems and Technology (TIST), Vol. 9, 4 (2018), 1--26.
[28]
FM Lord, MR Novick, and Allan Birnbaum. 1968. Statistical theories of mental test scores. (1968).
[29]
Yishay Mansour, Mehryar Mohri, Jae Ro, and Ananda Theertha Suresh. 2020. Three approaches for personalization with applications to federated learning. arXiv preprint arXiv:2002.10619 (2020).
[30]
H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. 2016. Communication-efficient learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629 (2016).
[31]
Kritphong Mongkhonvanit, Klint Kanopka, and David Lang. 2019. Deep Knowledge Tracing and Engagement with MOOCs. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge. ACM, 340--342.
[32]
Maryam M Najafabadi, Flavio Villanustre, Taghi M Khoshgoftaar, Naeem Seliya, Randall Wald, and Edin Muharemagic. 2015. Deep learning applications and challenges in big data analytics. Journal of Big Data, Vol. 2, 1 (2015), 1.
[33]
Hiromi Nakagawa, Yusuke Iwasawa, and Yutaka Matsuo. 2019. Graph-based Knowledge Tracing: Modeling Student Proficiency Using Graph Neural Network. In 2019 IEEE/WIC/ACM International Conference on Web Intelligence (WI). IEEE, 156--163.
[34]
Zachary A Pardos and Neil T Heffernan. 2011. KT-IDEM: introducing item difficulty to the knowledge tracing model. In International conference on user modeling, adaptation, and personalization. Springer, 243--254.
[35]
Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge tracing. In Advances in neural information processing systems. 505--513.
[36]
James J Ryan. 1968. Teacher judgments of test item properties. Journal of Educational Measurement, Vol. 5, 4 (1968), 301--306.
[37]
Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojciech Samek. 2019. Robust and communication-efficient federated learning from non-iid data. IEEE transactions on neural networks and learning systems (2019).
[38]
Jessica Sharkness and Linda DeAngelo. 2011. Measuring student involvement: A comparison of classical test theory and item response theory in the construction of scales from student surveys. Research in Higher Education, Vol. 52, 5 (2011), 480--507.
[39]
Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. 2020. Neural cognitive diagnosis for intelligent education systems. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 6153--6161.
[40]
Hao Wang, Tong Xu, Qi Liu, Defu Lian, Enhong Chen, Dongfang Du, Han Wu, and Wen Su. 2019. MCNE: An end-to-end framework for learning multiple conditional network representations of social network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1064--1072.
[41]
Lisa Wang, Angela Sy, Larry Liu, and Chris Piech. 2017. Deep knowledge tracing on programming exercises. In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale. 201--204.
[42]
Xiaolu Xiong, Siyuan Zhao, Eric G Van Inwegen, and Joseph E Beck. 2016. Going deeper with deep knowledge tracing. International Educational Data Mining Society (2016).
[43]
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019 a. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), Vol. 10, 2 (2019), 12.
[44]
Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Francc oise Beaufays. 2018. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02903 (2018).
[45]
Wensi Yang, Yuhang Zhang, Kejiang Ye, Li Li, and Cheng-Zhong Xu. 2019 b. FFD: A Federated Learning Based Method for Credit Card Fraud Detection. In International Conference on Big Data. Springer, 18--32.
[46]
Chun-Kit Yeung and Dit-Yan Yeung. 2018. Addressing two problems in deep knowledge tracing via prediction-consistent regularization. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale. 1--10.
[47]
Michael V Yudelson, Kenneth R Koedinger, and Geoffrey J Gordon. 2013. Individualized bayesian knowledge tracing models. In International conference on artificial intelligence in education. Springer, 171--180.
[48]
Jiani Zhang, Xingjian Shi, Irwin King, and Dit-Yan Yeung. 2017. Dynamic key-value memory networks for knowledge tracing. In Proceedings of the 26th international conference on World Wide Web. International World Wide Web Conferences Steering Committee, 765--774.
[49]
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. 2018. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582 (2018).

Cited By

View all
  • (2024)FDKT: Towards an Interpretable Deep Knowledge Tracing via Fuzzy ReasoningACM Transactions on Information Systems10.1145/365616742:5(1-26)Online publication date: 13-May-2024
  • (2024)Knowledge-Associated Embedding for Memory-Aware Knowledge TracingIEEE Transactions on Computational Social Systems10.1109/TCSS.2023.330690911:3(4016-4028)Online publication date: Jun-2024
  • (2023)Federated User Modeling from Hierarchical InformationACM Transactions on Information Systems10.1145/356048541:2(1-33)Online publication date: 9-Feb-2023
  • Show More Cited By

Index Terms

  1. Federated Deep Knowledge Tracing

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    WSDM '21: Proceedings of the 14th ACM International Conference on Web Search and Data Mining
    March 2021
    1192 pages
    ISBN:9781450382977
    DOI:10.1145/3437963
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 08 March 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. data isolation
    2. data quality evaluation
    3. federated learning
    4. intelligent education
    5. knowledge tracing

    Qualifiers

    • Research-article

    Funding Sources

    • CCF-Tencent Open Research Fund
    • National Key Research and Development Program of China
    • Foundation of State Key Laboratory of Cognitive Intelligence
    • National Natural Science Foundation of China

    Conference

    WSDM '21

    Acceptance Rates

    Overall Acceptance Rate 498 of 2,863 submissions, 17%

    Upcoming Conference

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)86
    • Downloads (Last 6 weeks)6
    Reflects downloads up to 21 Nov 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)FDKT: Towards an Interpretable Deep Knowledge Tracing via Fuzzy ReasoningACM Transactions on Information Systems10.1145/365616742:5(1-26)Online publication date: 13-May-2024
    • (2024)Knowledge-Associated Embedding for Memory-Aware Knowledge TracingIEEE Transactions on Computational Social Systems10.1109/TCSS.2023.330690911:3(4016-4028)Online publication date: Jun-2024
    • (2023)Federated User Modeling from Hierarchical InformationACM Transactions on Information Systems10.1145/356048541:2(1-33)Online publication date: 9-Feb-2023
    • (2022)PYKTProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3601617(18542-18555)Online publication date: 28-Nov-2022
    • (2022)Dynamic Clustering Federated Learning for Non-IID DataArtificial Intelligence10.1007/978-3-031-20503-3_10(119-131)Online publication date: 17-Dec-2022
    • (2022)Leveraging Artificial Intelligence Techniques for Effective Scaffolding of Personalized Learning in WorkplacesArtificial Intelligence Education in the Context of Work10.1007/978-3-031-14489-9_4(59-76)Online publication date: 29-Oct-2022
    • (2021)Time Saving Students’ Formative Assessment: Algorithm to Balance Number of Tasks and Result ReliabilityApplied Sciences10.3390/app1113604811:13(6048)Online publication date: 29-Jun-2021
    • (2021)A survey on federated learning in data miningWIREs Data Mining and Knowledge Discovery10.1002/widm.144312:1Online publication date: 9-Dec-2021

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media