Elamur Research⌕
☰
home / papers / 2022 / 深度知识追踪模型综述和性能比较

深度知识追踪模型综述和性能比较

Summary

Knowledge tracing (KT) infers how well a learner has mastered each knowledge concept from their record of exercise attempts, then uses that estimate to forecast their next answers. Classical approaches such as IRT, BKT and factor-analysis models (LFA, PFA, IFA) depend on hand-crafted features and fixed theoretical assumptions. As a result, they overlook links between concepts and the effects of forgetting. This Chinese-language survey from East China Normal University reviews the deep models that have largely replaced them since DKT (Piech et al., 2015).

The authors sort the models into four families by backbone: recurrent networks, memory networks, self-attention networks, and other designs such as CNN, GNN, federated and transfer learning. Within the first three families, they further group work by the kind of improvement: changing the architecture, adding extra interaction features, or building in educational theory. They then benchmark eight representative models on five public datasets. The paper closes with real teaching use cases and open research questions. It offers a structured map and an empirical baseline for anyone building learner models for adaptive tutoring.

Key points

  • Bibliographic note: the paper appeared in 软件学报 (Journal of Software) 34(3):1365–1395 in 2023, with DOI 10.13328/j.cnki.jos.006715. It was published online on 2022-07-22, which explains the "2022" date.
  • Recurrent-network models reviewed: stacked LSTM (Sha et al.), EKT, KQN, tree-based DKT, EERNN, DKTS, qDKT, GIKT, DKT-DSC, DKT-F, KPT and IEKT. Memory-network models trace back to Zhang et al., and self-attention models to Pandey et al.
  • Examples of added features:
    • EERNN encodes exercise text with a Bi-LSTM.
    • Tong et al. use BERT and TextCNN to derive difficulty and concept features.
    • DKT-F adds repeat-interval, sequence-interval and past-attempt-count inputs to model forgetting.
    • DKT-DSC periodically clusters students by ability with K-means.
  • Benchmark scope: 8 models are compared on 5 public datasets. The experimental section was missing from the supplied text, so dataset names and per-model scores are not recorded here.
  • Original DKT limitation: it treats each exercise as covering a single concept. It therefore cannot tell apart different exercises on the same concept, which motivated many of the later variants.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.13328/j.cnki.jos.006715

@article{2022,
  title = {{深度知识追踪模型综述和性能比较}},
  author = {王宇 and 朱梦霞 and 杨尚辉 and 陆雪松 and 周傲英},
  journal = {软件学报 (Journal of Software)},
  year = {2022},
  doi = {10.13328/j.cnki.jos.006715},
  url = {https://doi.org/10.13328/j.cnki.jos.006715}
}

References

78 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wolins L, Wright BD, Rasch G. Probabilistic models for some intelligence and attainment tests. Journal of the American Statis tical Association, 1982, 77(377): 220. find
  2. Corbett AT, Anderson JR. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modelling and User-adapted Interaction, 1995, 4(4): 253278. find
  3. Cen H, Koedinger K, Junker B. Learning factors analysis—A general method for cognitive model evaluation and improvement. In: Ikeda M, Ashley KD, Chan TW, eds. Proc. of the Intelligent Tutoring Systems, Vol.4053. Berlin, Heidelberg: Springer, 2006. 164175. find
  4. Pavlik PI, Cen H, Koedinger KR. Performance factors analysis—A new alternative to knowledge tracing. In: Proc. of the Conf. on Artificial Intelligence in Education: Building Learning Systems that Care: From Knowledge Representation to Affective Modelling. 2009. 531538. find
  5. Chi M, Koedinger K, Gordon G, Jordan P, VanLehn K. Instructional factors analysis: A cognitive model for multiple instructional interventions. In: Proc. of the 12th Int’l Conf. on Educational Data Mining. 2011. 6170. find
  6. Liu HY, Zhang TC, Wu PW, Yu G. A review of knowledge tracking. Journal of East China Normal University (Natural Science), 2019, 2019(5): 115 (in Chinese with English abstract). [doi: 10.3969/j.issn.1000-5641.2019.05.001] — [doi:10.3969/j.issn.1000-5641.2019.05.001](https://doi.org/10.3969/j.issn.1000-5641.2019.05.001) find
  7. Liu Q, Shen S, Huang Z, Chen E, Zheng Y. A survey of knowledge tracing. arXiv:2105.15106, 2021. — [arXiv:2105.15106](https://arxiv.org/abs/2105.15106) find
  8. Piech C, Spencer J, Huang J, Ganguli S, Sahami M, Guibas L, Sohl-Dickstein J. Deep knowledge tracing. In: Proc. of the Advances in Neural Information Processing Systems. Montreal, 2015. 505513. find
  9. Zhang J, Shi X, King I, Yeung DY. Dynamic key-value memory networks for knowledge tracing. In: Proc. of the 26th Int’l Conf. on World Wide Web. 2017. 765774. find
  10. Weston J, Chopra S, Bordes A. Memory networks. arXiv:1410.3916, 2015. find
  11. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. In: Proc. of the Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017 . 59986008. find
  12. Pandey S, Karypis G. A self-attentive model for knowledge tracing. In: Proc. of the 12th Int’l Conf. on Educational Data Mining. 2019. 384389. find
  13. Sha L, Hong P. Neural knowledge tracing. In: Proc. of the Int’l Conf. on Brain Function Assessment in Learning. Cham: Springer, 2017. 108117. find
  14. Liu Q, Huang Z, Yin Y, Chen E, Xiong H, Su Y, Hu G. EKT: Exercise-aware knowledge tracing for student performance prediction. IEEE Trans. on Knowledge and Data Engineering, 2019, 33(1): 100115. find
  15. Lee J, Yeung DY. Knowledge query network for knowledge tracing: How knowledge interacts with skills. In: Proc. of the 9th Int’l Conf. on Learning Analytics & Knowledge. Tempe: ACM, 2019. 491500. find
  16. Yang H, Cheung LP. Implicit heterogeneous features embedding in deep knowledge tracing. Cognitive Computation, 2018, 10(1): 314. find
  17. Zhang L, Xiong X, Zhao S, Botelho A, Heffernan NT. Incorporating rich features into deep knowledge tracing. In: Proc. of the 4th (2017) ACM Conf. on Learning @ Scale. Cambridge, Massachusetts: ACM, 2017. 169172. find
  18. Su Y, Liu Q, Liu Q, Huang Z, Yin Y, Chen E, Ding C, Wei S, Hu G. Exercise-enhanced sequential modeling for student performance prediction. In: Proc. of the AAAI Conf. on Artificial Intelligence. 2018. 24352443. find
  19. Wang Z, Feng X, Tang J, Huang GY, Liu Z. Deep knowledge tracing with side information. In: Proc. of the Int’l Conf. on Artificial Intelligence in Education. Cham: Springer, 2019. 303308. find
  20. Sonkar S, Waters AE, Lan AS, Grimaldi PJ, Baraniuk RG. QDKT: Question-centric deep knowledge tracing. In: Proc. of the Int’l Conf. on Artificial Intelligence in Education. Cham: Springer, 2021. 433437. find
  21. Liu D, Zhang Y, Zhang J, Li Q, Zhang C, Yin Y. Multiple features fusion attention mechanism enhanced deep knowledge tracing for student performance prediction. IEEE Access, 2020, 8: 194894194903. find
  22. Tong H, Zhou Y, Wang Z. Exercise hierarchical feature enhanced knowledge tracing. In: Proc. of the Int’l Conf. on Artificial Intelligence in Education. Cham: Springer, 2020. 324328. find
  23. Zhang N, Du Y, Deng K, Li L, Shen J, Sun G. Attention-based knowledge tracing with heterogeneous information network embedding. In: Li G, Shen HT, Yuan Y, Wang X, Liu H, Zhao X, eds. Proc. of the Knowledge Science, Engineering and Management, Vol.12274. Cham: Springer, 2020. 95103. find
  24. Yang Y, Shen J, Qu Y, Liu Y, Wang K, Zhu Y, Zhang W, Yu Y. GIKT: A graph-based interaction model for knowledge tracing. In: Proc. of the Joint European Conf. on Machine Learning and Knowledge Discovery in Databases. Cham: Springer, 2020. 299315. find
  25. Chen P, Lu Y, Zheng VW, Pian Y. Prerequisite-driven deep knowledge tracing. In: Proc. of the IEEE Int’l Conf. on Data Mining (ICDM). Singapore: IEEE, 2018. 3948. find
  26. Minn S, Yu Y, Desmarais MC, Zhu F, Vie JJ. Deep knowledge tracing and dynamic student classification for knowledge tracing. In: Proc. of the IEEE Int’l Conf. on Data Mining (ICDM). Qingdao: IEEE, 2018. 11821187. find
  27. Nagatani K, Zhang Q, Sato M, Chen YY, Chen F, Ohkuma T. Augmenting knowledge tracing by considering forgetting behavior. In: Proc. of the World Wide Web Conf. San Francisco: ACM, 2019. 31013107. find
  28. Ebbinghaus H. Memory: A contribution to experimental psychology. Annals of Neurosciences, 2013, 20(4): 155156. find
  29. Huang Z, Liu Q, Chen Y, Wu L, Xiao K, Chen E, Ma H, Hu G. Learning or forgetting? A dynamic approach for tracking the knowledge proficiency of students. ACM Trans. on Information Systems (TOIS), 2020, 38(2): 133. find
  30. Long T, Liu Y, Shen J, Zhang W, Yu Y. Tracing knowledge state with individual cognition and acquisition estimation. In: Proc. of the 44th Int’l ACM SIGIR Conf. on Research and Development in Information Retrieval. Virtual Event: ACM, 2021 . 173182. find
  31. Abdelrahman G, Wang Q. Knowledge tracing with sequential key-value memory networks. In: Proc. of the 42nd Int’l ACM SIGIR Conf. on Research and Development in Information Retrieval. 2019. 175184. find
  32. Chaudhry R, Singh H, Dogga P, Saini SK. Modeling Hint-taking Behavior and Knowledge State of Students with Multi-task Learning. Int’l Educational Data Mining Society, 2018. find
  33. Sun X, Zhao X, Ma Y, Yuan X, He F, Feng J. Muti-behavior features based knowledge tracking using decision tree improved DKVMN. In: Proc. of the ACM Turing Celebration Conf. Chengdu: ACM, 2019. 16. find
  34. Minn S, Desmarais MC, Zhu F, Xiao J, Wang J. Dynamic student classiffication on memory networks for knowledge tracing. In: Proc. of the Pacific-Asia Conf. on Knowledge Discovery and Data Mining. Cham: Springer, 2019. 163174. find
  35. Sun X, Zhao X, Li B, Ma Y, Sutcliffe R, Feng J. Dynamic key-value memory networks with rich features for knowledge tracing. IEEE Trans. on Cybernetics, 2022, 52(8): 8239–8245. find
  36. Ha H, Hwang U, Hong Y, Jang J, Yoon S. Deep trustworthy knowledge tracing. arXiv:1805.10768, 2018. — [arXiv:1805.10768](https://arxiv.org/abs/1805.10768) find
  37. Yeung CK. Deep-IRT: Make deep learning based knowledge tracing explainable using item response theory. arXiv:1904.11738, 2019. — [arXiv:1904.11738](https://arxiv.org/abs/1904.11738) find
  38. Gan W, Sun Y, Sun Y. Knowledge interaction enhanced knowledge tracing for learner performance prediction. In: Proc. of the 7th Int’l Conf. on Behavioural and Social Computing (BESC). Bournemouth: IEEE, 2020. 16. find
  39. Abdelrahman G, Wang Q. Deep graph memory networks for forgetting-robust knowledge tracing. arXiv:2108.08105, 2021. — [arXiv:2108.08105](https://arxiv.org/abs/2108.08105) find
  40. Choi Y, Lee Y, Cho J, Baek J, Kim B, Cha Y, Shin D, Bae C, Heo J. Towards an appropriate query, key, and value computation fo r knowledge tracing. In: Proc. of the 11th Int’l Learning Analytics and Knowledge Conf. (LAK 2021). 2021. 490496. find
  41. Bhatt S, Zhao J, Thille C, Zimmaro D, Gattani N. A novel approach for knowledge state representation and prediction. In: Proc. of the 7th ACM Conf. on Learning @ Scale. Virtual Event: ACM, 2020. 353356. find
  42. Zhang C, Jiang Y, Zhang W, Gu C. MUSE: Multi-scale temporal features evolution for knowledge tracing. arXiv:2102.00228, 2021. — [arXiv:2102.00228](https://arxiv.org/abs/2102.00228) find
  43. Oya T, Morishima S. LSTM-SAKT: LSTM-encoded SAKT-like transformer for knowledge tracing. arXiv:2102.00845, 2021. — [arXiv:2102.00845](https://arxiv.org/abs/2102.00845) find
  44. Ghosh A, Heffernan N, Lan AS. Context-aware attentive knowledge tracing. In: Proc. of the 26th ACM SIGKDD Int’l Conf. on Knowledge Discovery & Data Mining. 2020. 23302339. find
  45. Pandey S, Srivastava J. RKT: Relation-aware self-attention for knowledge tracing. In: Proc. of the 29th ACM Int’l Conf. on Information & Knowledge Management. 2020. 12051214. find
  46. Zhou Y, Li X, Cao Y, Zhao X, Ye Q, Lv J. LANA: Towards Personalized Deep Knowledge Tracing through Distinguishable Interactive Sequences. Int’l Educational Data Mining Society, 2021. find
  47. Shen S, Liu Q, Chen E, Wu H, Huang Z, Zhao W, Su Y, Ma H, Wang S. Convolutional knowledge tracing: modeling individualization in student learning process. In: Proc. of the 43rd Int’l ACM SIGIR Conf. on Research and Development in Information Retrieval. Virtual Event: ACM, 2020. 18571860. find
  48. Nakagawa H, Iwasawa Y, Matsuo Y. Graph-based knowledge tracing: modeling student proficiency using graph neural network. In: Proc. of the IEEE/WIC/ACM Int’l Conf. on Web Intelligence. Thessaloniki: ACM, 2019. 156163. find
  49. Cheng S, Liu Q, Chen E, Zhang K, Huang Z, Yin Y, Huang X, Su Y. AdaptKT: A domain adaptable method for knowledge tracing. In: Proc. of the 15th ACM Int’l Conf. on Web Search and Data Mining. Virtual Event: ACM, 2022. 123131. find
  50. Yeung CK, Yeung DY. Incorporating features learned by an enhanced deep knowledge tracing model for STEM/non-STEM job prediction. Int’l Journal of Artificial Intelligence in Education, 2019, 29(3): 317341. find
  51. Swamy V, Guo A, Lau S, Wu W, Wu M, Pardos Z, Culler D. Deep knowledge tracing for free-form student code progression. In: Penstein Rosé C, Martínez -Maldonado R, Hoppe HU, Luckin R, Mavrikis M, Porayska-Pomsta K, McLaren B, Du Boulay B, eds. Proc. of the Artificial Intelligence in Education, Vol.109… find
  52. Wang L, Sy A, Liu L, Piech C. Deep knowledge tracing on programming exercises. In: Proc. of the 4th (2017) ACM Conf. on Learning @ Scale. Cambridge Massachusetts: ACM, 2017. 201204. find
  53. Kaneko M, Kajiwara T, Komachi M. TMU system for SLAM-2018. In: Proc. of the 13th Workshop on Innovative Use of NLP for Building Educational Applications. New Orleans: Association for Computational Linguistics, 2018 . 365369. find
  54. Jiang W, Pardos ZA, Wei Q. Goal-based course recommendation. In: Proc. of the 9th Int’l Conf. on Learning Analytics & Knowledge. Tempe: ACM, 2019. 3645. find
  55. Zhang J, King I. Topological order discovery via deep knowledge tracing. In: Hirose A, Ozawa S, Doya K, Ikeda K, Lee M, Liu D, eds. Proc. of the Neural Information Processing, Vol.9950. Cham: Springer, 2016. 112119. find
  56. Zeiler MD, Fergus R. Visualizing and understanding convolutional networks. In: Proc. of the European Conf. on Computer Vision. Cham: Springer, 2014. 818833. find our card
  57. Nguyen A, Dosovitskiy A, Yosinski J, Brox T, Clune J. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In: Proc. of the Advances in Neural Information Processing Systems, Vol.29. 2016. 33873395. find
  58. Baehrens D, Schroeter T, Harmeling S, Kawanabe M, Hansen K. How to explain individua l classification decisions. The Journal of Machine Learning Research, 2010, 11(6): 18031831. find
  59. Sundararajan M, Taly A, Yan Q. Axiomatic attribution for deep networks. In: Proc. of the Int’l Conf. on Machine Learning. PMLR, 2017. 33193328. find
  60. Bastings J, Filippova K. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? In: Proc. of the 3rd BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. 2020. 149155. find
  61. Li J, Chen X, Hovy E, Jurafsky D. Visualizing and understanding neural models in NLP. In: Proc. of the NAACL-HLT. 2016. 681691. find our card
  62. Bach S, Binder A, Montavon G, Klauschen F, Müller KR, Samek W. On pixel -wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 2015, 10(7): Article No.0130140. find
  63. Arras L, Montavon G, Müller KR, Samek W. Explaining recurrent neural network predictions in sentiment analysis. In: Proc. of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. 2017. 159168. find
  64. Li J, Monroe W, Jurafsky D. Understanding neural networks through representation erasure. arXiv:1612.08220, 2016. — [arXiv:1612.08220](https://arxiv.org/abs/1612.08220) find
  65. DeYoung J, Jain S, Rajani NF, Lehman E, Xiong C, Socher R, Wallace BC. ERASER: A benchmark to evaluate rationalized NLP models. In: Proc. of the 58th Annual Meeting of the Association for Computational Linguistics. 2020. 44434458. find
  66. Jiang B, Wu S, Yin C, Zhang H. Knowledge tracing within single programming practice using problem -solving process data. IEEE Trans. on Learning Technologies, 2020, 13(4): 822832. find
  67. Renduchintala A, Koehn P, Eisner J. Knowledge tracing in sequential learning of inflected vocabulary. In: Proc. of the 21st Conf. on Computational Natural Language Learning (CoNLL 2017). Vancouver: Association for Computational Linguistics, 2017. 238247. find
  68. Zylich B, Lan A. Linguistic skill modeling for second language acquisition. In: Proc. of the LAK21: the 11th Int’l Learning Analytics and Knowledge Conf. Irvine: ACM, 2021. 141150. find
  69. Allamanis M, Brockschmidt M, Khademi M. Learning to represent programs with graphs. arXiv:1711.00740, 2017. find
  70. Zhang J, Wang X, Zhang H, Sun H, Wang K, Liu X. A novel neural source code representation based on abstract syntax tree. In: Proc. of the IEEE/ACM 41st Int’l Conf. on Software Engineering (ICSE). Montreal: IEEE, 2019. 783794. find
  71. Piech C, Huang J, Nguyen A, Phulsuksombati M, Sahami M, Guibas L. Learning program embeddings to propagate feedback on student code. In: Proc. of the Int’l Conf. on Machine Learning. PMLR, 2015. 10931102. find
  72. Wei H, Li H, Xia M, Wang Y, Qu H. Predicting student performance in interactive online question pools using mouse interaction features. In: Proc. of the 10th Int’l Conf. on Learning Analytics & Knowledge. 2020. 645654. find
  73. Chopade P, Khan SM, Edwards D, Von Davier A. Machine learning for efficient assessment and prediction of human performance in collaborative learning environments. In: Proc. of the IEEE Int’l Symp. on Technologies for Homeland Security (HST). Woburn: IEEE, 2018. 16. find
  74. Olsen JK, Sharma K, Rummel N, Aleven V. Temporal analysis of multimodal data to predict collaborative learning outcomes. British Journal of Educational Technology, 2020, 51(5): 15271547. find
  75. He X, Liao L, Zhang H, Nie L, Hu X, Chua TS. Neural collaborative filtering. In: Proc. of the 26th Int’l Conf. on World Wide Web. 2017. 173182. find
  76. Zhao J, Bhatt S, Thille C, Zimmaro D, Gattani N. Interpretable personalized knowledge tracing and next learning activity recommendation. In: Proc. of the 7th ACM Conf. on Learning @ Scale. Virtual Event: ACM, 2020. 325328. find
  77. Huang Z, Liu Q, Zhai C, Yin Y, Chen E, Gao W, Hu G. Exploring multi-objective exercise recommendations in online education systems. In: Proc. of the 28th ACM Int’l Conf. on Information and Knowledge Management. Beijing: ACM, 2019. 12611270. find
  78. Bassen J, Balaji B, Schaarschmidt M, Thille C, Painter J, Zimmaro D, Games A, Fast E, Mitchell JC. Reinforcement learning for the adaptive scheduling of educational activities. In: Proc. of the CHI Conf. on Human Factors in Computing Systems. Honolulu: ACM, 2020. 112. 附中文参考文献: [6] 刘恒宇, 张天成, 武培文, 于戈… — [doi:10.3969/j.issn.1000-5641.2019.05.001](https://doi.org/10.3969/j.issn.1000-5641.2019.05.001) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/wang-2022-performance-comparison-deep/