Elamur Research⌕
☰
home / papers / 2015 / Deep knowledge tracing

Deep knowledge tracing

Paper · 2015 · Piech

Summary

The paper tackles knowledge tracing: predicting whether a learner will answer the next exercise correctly, given their history of attempts and outcomes. Earlier methods, mainly Bayesian Knowledge Tracing (a Hidden Markov Model over binary per-skill mastery variables), rely on restrictive assumptions and on expert-assigned concept labels for each exercise. The authors propose Deep Knowledge Tracing (DKT), which uses recurrent networks (a vanilla RNN with sigmoid units and an LSTM) to learn a continuous, high-dimensional student state directly from interaction sequences. Each attempt is encoded as a one-hot vector of exercise and correctness, or as a compressed representation when there are many exercises. The output is a probability of success for every exercise. DKT is evaluated on simulated data, the Assistments benchmark and Khan Academy data, and it clearly beats prior state of the art. The trained network can also expose how exercises influence each other and can help order curricula without human annotation. This matters for scalable, personalized online tutoring.

Key points

  • The authors report roughly a 25% AUC improvement over the best earlier result on a knowledge tracing benchmark, with gains on both Assistments and Khan Academy data.
  • For datasets with M distinct exercises, each input is a one-hot vector of size 2M encoding exercise and outcome jointly. Encoding the two parts separately reduced accuracy.
  • On synthetic data with 5 hidden concepts, the learned influence graph clustered exercises perfectly by concept.
  • On Khan Academy 8th grade Common Core math, influence graphs (edges below 0.1 removed) formed coherent topic clusters. These clusters were absent from simple frequency-based baselines.
  • Limitations and plans: DKT needs large training sets, which suits online platforms but not small classrooms. Code is provided in the supplemental material, and a controlled curriculum experiment with Khan Academy is planned.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,462 citations🎓Stanford UniversityStanford · United States506 works · 330,052 citations

Also on the byline: Khan Academy.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksInformation Systems · 1,730 worksComputer Science Applications · 131 works

Cite

Cite as Piech. 2015. Deep knowledge tracing. arXiv (Cornell University).

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1506.05908

@misc{piech2015deep,
  title = {{Deep knowledge tracing}},
  author = {Piech},
  year = {2015},
  eprint = {1506.05908},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1506.05908}
}

References

33 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Khan academy privacy notice https://www.khanacademy.org/about/privacy-policy, 2015. ↗
  2. B ARANIUK , R. Compressive sensing. IEEE signal processing magazine 24, 4 (2007). find
  3. C EN , H., K OEDINGER , K., AND J UNKER , B. Learning factors analysis–a general method for cognitive model evaluation and improvement. In Intelligent tutoring systems (2006), Springer, pp. 164–175. find
  4. C OHEN , G. L., AND G ARCIA , J. Identity, belonging, and achievement a model, interventions, implications. Current Directions in Psychological Science 17, 6 (2008), 365–369. find
  5. C ORBETT , A. Cognitive computer tutors: Solving the two-sigma problem. In User Modeling 2001. Springer, 2001, pp. 137–147. 5 22 30 31 7 11 3 18 20 5 22 36 30 31 41 36 42 9 Linear models of bivariate data 32 Understand equations word problems 55 Pythagorean theorem 2 10 Systems of equations with eli… find
  6. C ORBETT , A. T., AND A NDERSON , J. R. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4, 4 (1994), 253–278. find
  7. D B AKER , R. S. J., C ORBETT , A. T., AND A LEVEN , V. More accurate student modeling through contextual estimation of slip and guess probabilities in bayesian knowledge tracing. In Intelligent Tutoring Systems (2008), Springer, pp. 406–415. find
  8. D B AKER , R. S. J., P ARDOS , Z. A., G OWDA , S. M., N OORAEI , B. B., AND H EFFERNAN , N. T. Ensembling predictions of student knowledge within intelligent tutoring systems. In User Modeling, Adaption and Personalization. Springer, 2011, pp. 13–24. find
  9. D RASGOW , F., AND H ULIN , C. L. Item response theory. Handbook of industrial and organizational psychology 1 (1990), 577–636. find
  10. E LLIOT , A. J., AND D WECK , C. S. Handbook of competence and motivation. Guilford Publications, 2013. find
  11. F ENG , M., H EFFERNAN , N., AND K OEDINGER , K. Addressing the assessment challenge with an online system that tutors as it assesses. User Modeling and User-Adapted Interaction 19, 3 (2009), 243–266. find
  12. F ITCH , W. T., H AUSER , M. D., AND C HOMSKY , N. The evolution of the language faculty: clarifications and implications. Cognition 97, 2 (2005), 179–210. find
  13. G ENTNER , D. Structure-mapping: A theoretical framework for analogy. Cognitive science 7, 2.
  14. G ONG , Y., B ECK , J. E., AND H EFFERNAN , N. T. In Intelligent Tutoring Systems, Springer. find
  15. G RAVES , A., M OHAMED , A.-R., AND H INTON , G. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on (2013), IEEE, pp. 6645–6649. find
  16. H OCHREITER , S., AND S CHMIDHUBER , J. Long short-term memory. Neural computation 9, 8. find
  17. K ARPATHY , A., AND F EI -F EI , L. Deep visual-semantic alignments for generating image descriptions. arXiv preprint arXiv:1412.2306 (2014).
  18. K HAJAH , M., W ING , R. M., L INDSEY , R. V., AND M OZER , M. C. Incorporating latent factors into knowledge tracing to predict individual differences in learning. Proceedings of the 7th International Conference on Educational Data Mining (2014). find
  19. K HAJAH , M. M., H UANG , Y., G ONZ ÁLEZ -B RENES , J. P., M OZER , M. C., AND B RUSILOVSKY , P. Integrating knowledge tracing and item response theory: A tale of two frameworks. Proceedings of the 4th International Workshop on Personalization Approaches in Learning Environments (2014). find
  20. L AN , A. S., S TUDER , C., AND B ARANIUK , R. G. Time-varying learning and content analytics via sparse factor analysis. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining (2014), ACM, pp. 452–461. find
  21. L INNENBRINK , E. A., AND P INTRICH , P. R. Role of affect in cognitive processing in academic contexts. Motivation, emotion, and cognition: Integrative perspectives on intellectual functioning and development (2004), 57–87. find
  22. M IKOLOV , T., K ARAFI ÁT , M., B URGET , L., C ERNOCK Ỳ , J., AND K HUDANPUR , S. Recurrent neural network based language model. In INTERSPEECH 2010, 11th Annual Conference of the International Speech Communication Association, Makuhari, Chiba, Japan, 2010 (2010), pp. 1045–1048. find
  23. P ARDOS , Z. A., AND H EFFERNAN , N. T. Kt-idem: Introducing item difficulty to the knowledge tracing model. In User Modeling, Adaption and Personalization. Springer, 2011, pp. 243–254. find
  24. P AVLIK J R , P. I., C EN , H., AND K OEDINGER , K. R. Performance Factors Analysis–A New Alternative to Knowledge Tracing. Online Submission (2009). find
  25. P IECH , C., H UANG , J., N GUYEN , A., P HULSUKSOMBATI , M., S AHAMI , M., AND G UIBAS , L. J. Learning program embeddings to propagate feedback on student code. CoRR abs/1505.05969 (2015). — [arXiv:1505.05969](https://arxiv.org/abs/1505.05969) find
  26. P IECH , C., S AHAMI , M., H UANG , J., AND G UIBAS , L. Autonomously generating hints by inferring problem solving policies. find
  27. P IECH , C., S AHAMI , M., K OLLER , D., C OOPER , S., AND B LIKSTEIN , P. Modeling how students learn to program. In Proceedings of the 43rd ACM symposium on Computer Science Education. find
  28. P OLSON , M. C., AND R ICHARDSON , J. J. Foundations of intelligent tutoring systems. Psychology Press, 2013. find
  29. R AFFERTY , A. N., B RUNSKILL , E., G RIFFITHS , T. L., AND S HAFTO , P. Faster teaching by POMDP planning. In Artificial intelligence in education (2011), Springer, pp. 280–287. find
  30. R OHRER , D. The effects of spacing and mixing practice problems. Journal for Research in Mathematics Education (2009), 4–17. find
  31. S CHRAAGEN , J. M., C HIPMAN , S. F., AND S HALIN , V. L. Cognitive task analysis. Psychology Press, 2000. find
  32. W ILLIAMS , R. J., AND Z IPSER , D. A learning algorithm for continually running fully recurrent neural networks. Neural computation 1, 2 (1989), 270–280. find
  33. Y UDELSON , M. V., K OEDINGER , K. R., AND G ORDON , G. J. Individualized bayesian knowledge tracing models. In Artificial Intelligence in Education (2013), Springer, pp. 171–180. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/piech-2015-deep-knowledge-tracing/