Elamur Research⌕
☰
home / papers / 2016 / How deep is knowledge tracing?

How deep is knowledge tracing?

Paper · 2016 · Khajah

Summary

Khajah, Lindsey and Mozer investigate why deep knowledge tracing (DKT), an LSTM-based recurrent network for predicting whether a student will answer the next exercise correctly, was reported to beat Bayesian knowledge tracing (BKT), the field's standard hidden Markov model, by a wide margin. They did not dissect DKT. Instead, they proposed four kinds of structure in student data that classic BKT ignores: recency effects, the ordering of trials across skills, similarity between skills, and differences in individual ability. They then gave BKT extensions already described in the literature: forgetting, latent student abilities and automatic skill induction. They compared the extended BKT with DKT on four data sets: Assistments, Synthetic, Spanish and Statics. On average, the extended BKT matched DKT. Part of DKT's originally reported advantage appears to come from inconsistent evaluation. The authors conclude that DKT gains its edge from flexibility, not from discovering new representations. Interpretable, shallow models can therefore compete with deep ones in knowledge tracing.

Key points

  • Re-running BKT gave much higher AUCs than those reported in the original DKT paper (Piech et al.): 0.73 vs 0.67 on Assistments and 0.62 vs 0.54 on Synthetic. This points to an evaluation bias against BKT.
  • The models differ greatly in size. With 200 LSTM units and 50 skills, DKT has roughly 250,000 parameters, while BKT needs about 200.
  • On a scaled AUC measure, the enhanced BKT and DKT traded wins:
    • Enhanced BKT beat DKT by 20.0% on Synthetic and 3.0% on Spanish.
    • It trailed DKT by 8.3% on Assistments and 3.5% on Statics.
  • Each data set benefited most from a different extension:
    • Forgetting helped most on Assistments.
    • Skill discovery helped most on Synthetic.
    • Latent student ability helped most on Statics.
  • Forgetting and ability are cheap to add, whereas skill discovery requires costly MCMC inference. In exchange, the extended BKT keeps psychologically meaningful parameters and produces an explicit mapping of exercises to skills.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Colorado BoulderBoulder · United States76 works · 51,396 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksMathematics · 233 worksComputer Science Applications · 184 worksStatistics and Probability · 23 works

Cite

Cite as Khajah. 2016. How deep is knowledge tracing?. arXiv (Cornell University).

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1604.02416

@misc{khajah2016how,
  title = {{How deep is knowledge tracing?}},
  author = {Khajah},
  year = {2016},
  eprint = {1604.02416},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1604.02416}
}

References

31 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Atkinson and J. A. Paulson. An approach to the psychology of instruction. Psychology Bulletin, 78:49–61, 1972. find
  2. R. S. Baker, A. T. Corbett, and V. Aleven. More accurate student modeling through contextual estimation of slip and guess probabilities in Bayesian knowledge tracing. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems, pages 406–415, Berlin, Heidelberg, 2008. Springer… find
  3. A. T. Corbett and J. R. Anderson. Knowledge tracing: Modelling the acquisition of procedural knowledge. User Model. User-Adapt. Interact., 4(4):253–278, 1995. 4 Of course, the skill discovery mechanism we incorporated certainly does regroup exercises to form skills, but the form of this regrouping o… find
  4. P. De Boeck and M. Wilson. Explanatory Item Response Models: a Generalized Linear and Nonlinear Approach. Springer-Verlag, New York, NY, 2004. find
  5. A. Galyardt and I. Goldin. Move your lamp post: Recent data reflects learner knowledge better than older data. JEDM-Journal of Educational Data Mining, 7(2):83–108, 2015. find
  6. Z. Ghahramani and M. I. Jordan. Factorial hidden markov models. In D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, editors, Advances in Neural Information Processing Systems 8, pages 472–478. MIT Press, 1996. find
  7. R. Golden. How to optimize student learning using recurrent neural networks (educational technology). Web page, 2016. http://tinyurl.com/GoldenDKT, retrieved February 29, 2016. ↗
  8. J. P. Gonzales-Brenes. Modeling skill acquisition over time with sequence and topic modeling. In S. V. N. V. G. Lebanon, editor, Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics. JMLR, 2015. find
  9. K. Gregor, I. Danihelka, A. Graves, D. Rezende, and D. Wierstra. DRAW: A recurrent neural network for image generation. In Proceedings of the 32nd International Conference on Machine Learning, pages 1462–1471, 2015.
  10. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  11. H. Ishwaran and L. F. James. Generalized weighted chinese restaurant processes for species sampling mixture models. Statistica Sinica, pages 1211–1235, 2003. find
  12. M. Jones, T. Curran, M. C. Mozer, and M. H. Wilder. Sequential effects in response time reveal learning mechanisms and event representations. Psychological review, 120:628–666, 2013. find
  13. M. Khajah, Y. Huang, J. P. Gonzales-Brenes, M. C. Mozer, and P. Brusilovsky. Integrating knowledge tracing and item response theory: A tale of two frameworks. In M. Kravcik, O. C. Santos, and J. G. Boticario, editors, Proceedings of the 4th International Workshop on Personalization Approaches in Lea… find
  14. M. Khajah, R. M. Wing, R. V. Lindsey, and M. C. Mozer. Incorporating latent factors into knowledge tracing to predict individual differences in learning. In J. Stamper, Z. Pardos, M. Mavrikis, and B. M. McLaren, editors, Proceedings of the 7th International Conference on Educational Data Mining, pag… find
  15. K. Koedinger, R. Baker, K. Cunningham, A. Skogsholm, B. Leber, and J. Stamper. A data repository for the EDM community: The PSLC DataShop. In C. Romero, S. Ventura, M. Pechenizkiy, and R. Baker, editors, Handbook of Educ. Data Mining, http://pslcdatashop.org, 2010. ↗
  16. Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521:436–444, 2015. find
  17. R. Lindsey, J. Shroyer, H. Pashler, and M. Mozer. Improving student’s long-term knowledge retention with personalized review. Psychological Science, 25:639–47, 2014. find
  18. R. V. Lindsey, M. Khajah, and M. C. Mozer. Automatic discovery of cognitive skills to improve the prediction of student learning. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 1386–1394. Curran Asso… find
  19. V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, et al. Human-level control through deep reinforcement learning. Nature, 518:529–533, 2015. find
  20. M. C. Mozer. Induction of multiscale temporal structure. In J. E. Moody, S. J. Hanson, and R. P. Lippmann, editors, Advances in Neural Information Processing Systems 4, pages 275–282. Morgan-Kaufmann, 1992. find
  21. Z. A. Pardos and N. T. Heffernan. KT-IDEM: Introducing item difficulty to the knowledge tracing model. In User Modeling, Adaption and Personalization, pages 243–254. Springer, 2011. find
  22. C. Piech, J. Bassen, J. Huang, S. Ganguli, M. Sahami, L. J. Guibas, and J. Sohl-Dickstein. Deep knowledge tracing. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 505–513. Curran Associates, Inc., 2015.
  23. Y. Qiu, Y. Qi, H. Lu, Z. A. Pardos, and N. T. Heffernan. Does time matter? modeling the effect of time with Bayesian knowledge tracing. In M. Pechenizkiy, T. Calders, C. Conati, S. Ventura, C. Romero, and J. C. Stamper, editors, Educational Data Mining 2011, pages 139–148. www.educationaldatamining.… find
  24. D. Rohrer, R. F. Dedrick, and K. Burgess. The benefit of interleaved mathematics practice is not limited to superficially similar kinds of problems. Psychonomic Bulletin and Review, 21:1323–1330, 2014. find
  25. D. Rohrer, R. F. Dedrick, and S. Stershic. Interleaved practice improves mathematics learning. Journal of Educational Psychology, 107:900–908, 2015. find
  26. J. Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61:85–117, 2015. find
  27. D. Sliver, A. Huang, C. J. Maddison, A. Guez, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529:484–489, 2016. find
  28. P. Steif and N. Bier. OLI Engineering Statics – Fall 2011. Feb. 2014. find
  29. O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In Computer Vision and Pattern Recognition, 2015.
  30. H.-F. Yu and Others. Feature engineering and classifier ensemble for KDD cup 2010. Technical report, Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan, 2010. find
  31. M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, editors, Computer Vision – ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I, pages 818–833. Springer Internation…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khajah-2016-deep-knowledge-tracing/