Elamur Research⌕
☰
home / papers / 2006 / Learning Factors Analysis – A General Method for Cognitive Model Evaluation and Improvement

Learning Factors Analysis – A General Method for Cognitive Model Evaluation and Improvement

Paper · 2006 · Hao Cen, Kenneth Koedinger and Brian Junker

Summary

Intelligent tutors rely on cognitive models, which are sets of production rules or skills describing how learners solve problems. These models are usually designed by hand by domain experts, cognitive scientists and programmers, and such first versions often miss important distinctions. The authors propose Learning Factors Analysis (LFA), a semi-automated way to assess and refine these models using student-tutor logs. LFA turns the power-law learning curve into a logistic regression. The regression gives each student an intercept and gives each skill both an intercept and a learning slope. Experts then define difficulty factors that can split skills, and an A* search guided by AIC or BIC explores the resulting candidate models. The authors tested LFA on log data collected from the Geometry Cognitive Tutor's unit covering area. It produced models that scored better than the original 15-skill model and were still interpretable. Some of the original skill distinctions were confirmed, while others, such as keeping circumference and diameter as separate skills, looked unnecessary. The fitted parameters also show which skills students practise too much or too little, giving concrete guidance for adjusting the curriculum.

Key points

  • Dataset: 4,102 first-attempt actions from 24 students across 115 problem steps. The original cognitive model has 15 skills.
  • Search method: Four expert-defined factors (Embed, Backward, Repeat, FigurePart) feed a Binary Split operator. An A* search, capped at 50 node expansions, is run once with AIC and once with BIC as the heuristic.
  • Over-practice: Parallelogram-side starts at about .94 success probability yet gets roughly 14.9 practice opportunities per student.
  • Under-practice: Trapezoid-height rises only from .29 to .69 after about 4.2 opportunities, so it needs more practice.
  • Hidden weak sub-skill and runtime: Compose-by-multiplication appears mastered overall (final .92). After splitting, however, its segment variant reaches only .60 with fewer than 2 practices on average. The Java implementation scores one model in seconds and searches hundreds of models in 4–5 hours.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States633 works · 195,218 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksPsychology · 1,673 worksExperimental and Cognitive Psychology · 950 worksDevelopmental and Educational Psychology · 533 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,394 works

Cite

Cite as Hao Cen, Kenneth Koedinger and Brian Junker. 2006. Learning Factors Analysis – A General Method for Cognitive Model Evaluation and Improvement. Lecture Notes in Computer Science.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1007/11774303_17

@inproceedings{cen2006learning,
  title = {{Learning Factors Analysis – A General Method for Cognitive Model Evaluation and Improvement}},
  author = {Hao Cen, Kenneth Koedinger and Brian Junker},
  booktitle = {Lecture Notes in Computer Science},
  year = {2006},
  doi = {10.1007/11774303\_17},
  url = {https://doi.org/10.1007/11774303_17}
}

References

17 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baffes P., Mooney R J.: A Novel Application of Theory Refinement to Student Modeling. In: Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI-96). Portland OR (1996) 403-408 find
  2. Barnes T.: The Q-matrix Method: Mining Student Response Data for Knowledge. In: Proceedings of AAAI 2005 Educational Data Mining Workshop. (2005) find
  3. Cen, H, Koedinger, K, Junker B.: Automating Cognitive Model Improvement by A*Search and Logistic Regression. In: Proceedings of AAAI 2005 Educational Data Mining Workshop. (2005) find
  4. Corbett A.T., Anderson, J.R., O’Brien A.T.: Student Modelling in the ACT Programming Tutor. In: Cognitively Diagnostic Assessment. Erlbaum, Hillsdale, NJ (1995) find
  5. Corbett, A.T. and Anderson, J.R.: Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4. (1995) 253-278. find
  6. Croteau E.A., Heffernan N. T., Koedinger K.R.: Why are Algebra Word Problem Difficult? Using Tutoring Log Files and the Power Law of Learning to Select the Best Fitting Cognitive Model. In: Proceedings of Intelligent Tutoring Systems. (2004) find
  7. Draney, K., Pirolli, P., & Wilson, M.: A Measurement Model for a Complex Cognitive Skill. In: Cognitively Diagnostic Assessment. Erlbaum, Hillsdale, NJ (1995) find
  8. Junker B.W., Koedinger, K., Trottini M.: Finding Improvements in Student Models for Intelligent Tutoring Systems via Variable Selection for a Linear Logistic Test Model., presented at annual meeting of Psychometric Society Vancouver BC (2000) find
  9. Koedinger K. R.: http://pact.cs.cmu.edu/koedinger/koedingerResearch.html. (2000) ↗
  10. Koedinger K. R., Mathan S.: Distinguishing Qualitatively Different Kinds of Leaning Using Log Files and Learning Curves. In: Proc. of Intelligent Tutoring Systems (2004) find
  11. Koedinger, K. R., Nathan, M. J.: The Real Story Behind Story Problems: Effects Of Representations On Quantitative Reasoning. The Journal of the Learning Sciences, 13 (2). (2004) 129-164. find
  12. Koedinger K.R., Anderson, J.R., Hadley W. H., Mark M. A.: Intelligent Tutoring Goes to School in the Big City. In: Proceedings of the 7th World Conference on Art, Intelligence and Education, AACE. (1995) find
  13. Newell, A., Rosenbloom, P.: Mechanisms of Skill Acquisition and the Law of Practice. In Anderson J. (ed.): Cognitive Skills and Their Acquisition, Erlbaum Hillsdale NJ (1981) find
  14. Russell S., Norvig P., Artificial Intelligence, 2 nd edn. Prentice Hall (2003) find
  15. Tatsuoka, K.: Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement. 20(4) (1983) 345-354. find
  16. Ur, S., VanLehn, K.: STEPS: A Simulated, Tutorable Physics Student. Journal of Artificial Intelligence in Education. 6(4) (1995) 405-437 find
  17. Wasserman, L.: All of Statistics: A Concise Course in Statistical Inference. Springer (2004) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cen-2006-learning-factors-analysis-general-method/