Elamur Research⌕
☰
home / papers / 2019 / A self-attentive model for knowledge tracing

A self-attentive model for knowledge tracing

Paper · 2019 · Pandey

Summary

This paper addresses knowledge tracing: estimating how well a learner has mastered knowledge concepts (KCs) by predicting whether they will answer the next exercise correctly. Earlier RNN-based approaches such as DKT and DKVMN generalize poorly on sparse real-world interaction logs, where each student touches only a small share of the KCs. Pandey and Karypis propose SAKT (Self Attentive Knowledge Tracing), a Transformer-style model that uses no recurrence. Past exercise–answer pairs are embedded as keys and values, and the exercise being predicted serves as the query. Causal scaled dot-product attention selects the few earlier interactions most relevant to the current exercise. The architecture adds learned positional embeddings, multiple heads, a feed-forward layer, residual connections, layer normalization and a sigmoid output, and is trained with cross-entropy. The authors evaluate SAKT on one synthetic and four real-world datasets. It beats prior state-of-the-art models by 4.43% AUC on average and trains roughly an order of magnitude faster. The learned attention weights also reveal interpretable relations between exercises.

Key points

  • Data: Synthetic (4,000 simulated students, 50 exercises from 5 hidden concepts), ASSIST2009 (skill-builder, density 0.06), ASSIST2015 (19,917 students, 708,631 interactions, 100 skills, density 0.05), ASSISTChall and STATICS.
  • Input encoding: each interaction is encoded as a single index, e + r×E, which gives 2E possible values. Sequences are left-padded or split into chunks of length n.
  • Interpretability: on Synthetic, attention weights cluster exercises perfectly by their hidden concept, even linking exercises that sit far apart in the sequence (e.g., exercise 22 to exercise 5).
  • Ablation (d = 200, 5 heads by default): removing the attention block hurts the most. A single head is consistently worse. A second block brings no gain. Residual connections matter little, and dropping them slightly improves ASSIST2015.
  • Speed (NVIDIA Titan V): SAKT takes 1.4 s per epoch, compared with 65 s for DKT+, 45 s for DKT and 26 s for DKVMN.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MinnesotaMinneapolis · United States29 works · 2,234 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

Cite

Cite as Pandey. 2019. A self-attentive model for knowledge tracing. arXiv (Cornell University).

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1907.06837

@misc{pandey2019self,
  title = {{A self-attentive model for knowledge tracing}},
  author = {Pandey},
  year = {2019},
  eprint = {1907.06837},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.06837}
}

References

11 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint Figure 4: Training Efficiency on ASSIST2009 dataset. arXiv:1607.06450 (2016).
  2. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  3. Wang-Cheng Kang and Julian McAuley. 2018. Self-Attentive Sequential Recommendation. CoRR abs/1808.09781 (2018). — [arXiv:1808.09781](https://arxiv.org/abs/1808.09781) find
  4. Mohammad Khajah, Robert V Lindsey, and Michael C Mozer. 2016. How deep is knowledge tracing? arXiv preprint arXiv:1604.02416 (2016).
  5. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
  6. Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge tracing. In Advances in Neural Information Processing Systems. 505–513.
  7. Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. 2016. One-shot learning with memory-augmented neural networks. arXiv preprint arXiv:1605.06065 (2016). find
  8. John Self. 1990. Theoretical foundations for intelligent tutoring systems. Journal of Artificial Intelligence in Education 1, 4 (1990), 3–14. find
  9. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 5998–6008.
  10. Chun-Kit Yeung and Dit-Yan Yeung. 2018. Addressing two problems in deep knowledge tracing via prediction-consistent regularization. arXiv preprint arXiv:1806.02180 (2018). — [arXiv:1806.02180](https://arxiv.org/abs/1806.02180) find
  11. Jiani Zhang, Xingjian Shi, Irwin King, and Dit-Yan Yeung. Dynamic key-value memory networks for knowledge tracing. In Proceedings of the 26th International Conference on World Wide Web.

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pandey-2019-self-attentive-model-knowledge-tracing/