Elamur Research⌕
☰
home / papers / 2020 / Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing

Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing

Paper · 2020 · Youngduck Choi, Youngnam Lee, Junghyun Cho et al.

Summary

Knowledge tracing estimates how well a learner has mastered material from their past activity, typically as the chance of answering the next exercise correctly. The authors identify two weaknesses in earlier attention-based tracers (EKT, NPA, SAKT). These models use only one shallow attention layer, and they all build queries from exercises and keys/values from exercise-response interactions without testing alternatives. The authors propose SAINT (Separated Self-AttentIve Neural Knowledge Tracing), a Transformer encoder-decoder. The encoder applies stacked self-attention to exercise embeddings only. The decoder takes response embeddings and alternates self-attention with attention over the encoder output. Keeping the two streams apart lets the model stack many attention layers without losing accuracy. They tested SAINT on EdNet, a large dataset from the Santa mobile learning app. Compared with SAKT, SAINT raised AUC by 1.8%. It also outperformed several Transformer-based variants. The study shows that the choice of inputs for queries, keys and values matters a great deal in deep attention models for education.

Key points

  • Dataset: EdNet (from the Santa app) contains 627,347 users, 72,907,005 responses and 16,175 exercises, collected since 2016.
  • Best configuration: SAINT with 4 stacked blocks and d_model = 512 reached ACC 0.7368 and AUC 0.7811. Among the compared architectures (SAINT, LTMTI, UTMTI, SSAKT), LTMTI came closest.
  • Attention patterns: In the first decoder block, attention is sparse. In the last block, it is spread more evenly across past responses, which suggests deeper layers model richer relations.
  • Embedding ablation: A simple response embedding (position plus correctness) performed slightly better than one that also used exercise category, timestamp and elapsed time.
  • Prior baselines: The previous models compared against are EKT, NPA (Bi-LSTM with attention) and SAKT (single-layer Transformer).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States243 works · 251,774 citations🎓University of MichiganAnn Arbor · United States94 works · 18,548 citations🎓Yale UniversityNew Haven · United States112 works · 14,009 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

Cite

Cite as Youngduck Choi, Youngnam Lee, Junghyun Cho, Jineon Baek, Byungsoo Kim, Yeongmin Cha, Dongmin Shin, Chan Bae and Jaewe Heo. 2020. Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing. Proceedings of the Seventh ACM Conference on Learning @ Scale.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3386527.3405945 · arXiv 2002.07033

@inproceedings{choi2020appropriate,
  title = {{Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing}},
  author = {Youngduck Choi, Youngnam Lee, Junghyun Cho, Jineon Baek, Byungsoo Kim, Yeongmin Cha, Dongmin Shin, Chan Bae and Jaewe Heo},
  booktitle = {Proceedings of the Seventh ACM Conference on Learning @ Scale},
  year = {2020},
  doi = {10.1145/3386527.3405945},
  eprint = {2002.07033},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3386527.3405945}
}

References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).
  2. Youngduck Choi, Youngnam Lee, Dongmin Shin, Junghyun Cho, Seoyon Park, Seewoo Lee, Jineon Baek, Chan Bae, Byungsoo Kim, and Jaewe Heo. 2019. EdNet: A Large-Scale Hierarchical Dataset in Education. (2019). find
  3. Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4, 4 (1994), 253–278. find
  4. Ryan SJ d Baker, Albert T Corbett, and Vincent Aleven. 2008. More accurate student modeling through contextual estimation of slip and guess probabilities in bayesian knowledge tracing. In International conference on intelligent tutoring systems. Springer, 406–415. find
  5. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. 249–256. find
  6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  7. Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web. International World Wide Web Conferences Steering Committee, 173–182. find
  8. Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, Guoping Hu, and others. 2019. EKT: Exercise-aware Knowledge Tracing for Student Performance Prediction. IEEE Transactions on Knowledge and Data Engineering (2019). find
  9. Jussi Kasurinen and Uolevi Nikula. 2009. Estimating programming knowledge with Bayesian knowledge tracing. ACM SIGCSE Bulletin 41, 3 (2009), 313–317. find
  10. Diederik P Kingma and Jimmy Ba. 2014. ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION. arXiv preprint arXiv:1412.6980 (2014).
  11. Kangwook Lee, Jichan Chung, Yeongmin Cha, and Changho Suh. 2016. Machine Learning Approaches for Learning Analytics: Collaborative Filtering Or Regression With Experts?. In NIPS Workshop, Dec. 1–11. find
  12. Youngnam Lee, Youngduck Choi, Junghyun Cho, Alexander R Fabbri, Hyunbin Loh, Chanyou Hwang, Yongku Lee, Sang-Wook Kim, and Dragomir Radev. 2019. Creating A Neural Pedagogical Agent by Jointly Learning to Review and Assess. arXiv preprint arXiv:1906.10910 (2019). — [arXiv:1906.10910](https://arxiv.org/abs/1906.10910) find
  13. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. arXiv preprint arXiv:1508.04025 (2015).
  14. Shalini Pandey and George Karypis. 2019. A Self-Attentive model for Knowledge Tracing. arXiv preprint arXiv:1907.06837 (2019).
  15. Zachary Pardos and Neil Heffernan. 2010a. Navigating the parameter space of Bayesian Knowledge Tracing models: Visualizations of the convergence of the Expectation Maximization algorithm. In Educational Data Mining 2010. find
  16. Zachary A Pardos, Yoav Bergner, Daniel T Seaton, and David E Pritchard. 2013. Adapting Bayesian Knowledge Tracing to a Massive Open Online Course in edX. EDM 13 (2013), 137–144. find
  17. Zachary A Pardos and Neil T Heffernan. 2010b. Modeling individualization in a bayesian networks implementation of knowledge tracing. In International Conference on User Modeling, Adaptation, and Personalization. Springer, 255–266. find
  18. Yumeng Qiu, Yingmei Qi, Hanyuan Lu, Zachary A Pardos, and Neil T Heffernan. 2011. Does Time Matter? Modeling the Effect of Time with Bayesian Knowledge Tracing.. In EDM. 139–148. find
  19. Michael Sao Pedro, Ryan Baker, and Janice Gobert. 2013. Incorporating scaffolding and tutor context into bayesian knowledge tracing to predict inquiry skill acquisition. In Educational Data Mining 2013. find
  20. Nguyen Thai-Nghe, Lucas Drumond, Artus Krohn-Grimberghe, and Lars Schmidt-Thieme. 2010. Recommender system for predicting student performance. Procedia Computer Science 1, 2 (2010), 2811–2819. find
  21. Brett van De Sande. 2013. Properties of the bayesian knowledge tracing model. Journal of Educational Data Mining 5, 2 (2013), 1. find
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998–6008.
  23. Michael V Yudelson, Kenneth R Koedinger, and Geoffrey J Gordon. 2013. Individualized bayesian knowledge tracing models. In International conference on artificial intelligence in education. Springer, 171–180. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/choi-2020-appropriate-query-key-value-computation/