Elamur Research⌕
☰
home / papers / 2016 / Going deeper with deep knowledge tracing

Going deeper with deep knowledge tracing

Paper · 2016 · Xiong

Summary

This paper re-examines Deep Knowledge Tracing (DKT), a recurrent neural network method for predicting whether students will answer intelligent-tutoring problems correctly. DKT had been reported to beat classic student models by a wide margin. The WPI authors rebuilt DKT in Python with Google's TensorFlow, feeding an LSTM-based network one-hot vectors that encode skill and correctness. They compared it with Bayesian Knowledge Tracing (BKT) and Performance Factors Analysis (PFA). They repeated the original experiment on the ASSISTments 2009-2010 skill-builder data and added an ASSISTments 2014-2015 dataset and a KDD Cup 2010 dataset.

Inspecting the data, they found three factors that inflated DKT's earlier score:

  • randomly duplicated rows;
  • multi-skill responses repeated once per skill, which an RNN can exploit by recalling exact sequence patterns;
  • scaffolding questions that BKT and PFA did not use.

With cleaned data limited to main questions, DKT clearly beats BKT but shows no overall advantage over PFA. The work serves as a reproducibility check and a warning that preprocessing choices and data glitches can create apparent algorithmic gains.

Key points

  • The original Lua/Torch DKT reported an AUC of 0.86 on ASSISTments 2009-2010, versus 0.67 for BKT. That dataset has 525,535 response rows, 4,217 students and 124 skills.
  • The three issues found were duplicated records, inconsistent multi-skill tagging (rows repeated per skill) and the inclusion of scaffolding questions ignored by BKT and PFA.
  • The TensorFlow DKT uses 200 hidden units, mini-batch SGD with batches of 100 students, and dropout on the hidden layer. Code is at github.com/siyuanzhao/2016-EDM.
  • On properly prepared data, DKT performs roughly on par with PFA and substantially better than BKT, while using only response sequences.
  • Proposed future work includes a context-dependent DKT with student and content features, and applying the approach to wheel spinning, mastery speed and affect detection.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Worcester Polytechnic InstituteWorcester · United States14 works · 1,103 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 worksComputer Science Applications · 184 works

Cite

Cite as Xiong. 2016. Going deeper with deep knowledge tracing. Educational Data Mining.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{xiong2016going,
  title = {{Going deeper with deep knowledge tracing}},
  author = {Xiong},
  booktitle = {Educational Data Mining},
  year = {2016},
  url = {https://eric.ed.gov/?id=ED592679}
}

References

16 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Piech, C., Bassen, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L. J., & Sohl-Dickstein, J. (2015). Deep Knowledge Tracing. In Advances in Neural Information Processing Systems (pp. 505-513).
  2. Mikolov, T., & Zweig, G. (2012, July). Context dependent recurrent neural network language model. In SLT (pp. 234-239). find
  3. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., & Ghemawat, S. (2015). TensorFlow: Large-scale machine learning on heterogeneous systems. White paper, Google Research. find
  4. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929-1958.
  5. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. find
  6. Stamper, J., Niculescu-Mizil, A., Ritter, S., Gordon, G.J., & Koedinger, K.R. (2010). Algebra I 2005-2006. Challenge data set from KDD Cup 2010 Educational Data Mining Challenge. Find it at http://pslcdatashop.web.cmu.edu/KDDCup/downloads.jsp ↗
  7. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., ... & Dieleman, S. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484-489. find
  8. Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009, June). Curriculum learning. In Proceedings of the 26th annual international conference on machine learning (pp. 41-48). ACM.
  9. Pavlik Jr, P. I., Cen, H., & Koedinger, K. R. (2009). Performance Factors Analysis-A New Alternative to Knowledge Tracing. Online Submission. find
  10. Corbett, A. T., & Anderson, J. R. (1994). Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, 4(4), 253-278. find
  11. Gong, Y., Beck, J. E., & Heffernan, N. T. (2010, June). Comparing knowledge tracing and performance factor analysis by using multiple model fitting procedures. In Intelligent tutoring systems (pp. 35-44). Springer Berlin Heidelberg. find
  12. Heathcote, A., Brown, S., & Mewhort, D. J. K. (2000). The power law repealed: The case for an exponential law of practice. Psychonomic bulletin & review, 7(2), 185-207. find
  13. Cen, H., Koedinger, K., & Junker, B. (2006, June). Learning factors analysis–a general method for cognitive model evaluation and improvement. In Intelligent tutoring systems (pp. 164-175). Springer Berlin Heidelberg. — Learning Factors Analysis – A General Method for Cognitive Model Evaluation and Improvement
  14. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780. find
  15. Pardos, Z. A., & Heffernan, N. T. (2011). KT-IDEM: Introducing item difficulty to the knowledge tracing model. In User Modeling, Adaption and Personalization (pp. 243-254). Springer Berlin Heidelberg. find
  16. ASSISTments Data. (2015). Retrieved March 07, 2016, from https://sites.google.com/site/assistmentsdata/home/assistment -2009-2010-data/skill-builder-data-2009-2010 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiong-2016-going-deeper-deep-knowledge-tracing/