Elamur Research⌕
☰
home / papers / 2026 / Measuring Optimal Challenge: Trajectory-Based Difficulty Alignment in Open-Ended Language Tutoring

Measuring Optimal Challenge: Trajectory-Based Difficulty Alignment in Open-Ended Language Tutoring

Paper · 2026 · Ziqi Shu, Shuman Wang and Michael Hardy

Summary

In conversational EFL tutoring, tasks are improvised on the spot by a human tutor or an LLM rather than taken from a calibrated item bank. Item Response Theory, knowledge tracing and LLM-based difficulty estimators therefore cannot check whether a given task fit the learner at that moment. The authors infer fit from what happens in the chat after an exercise is posted. Each following turn is coded by speaker and function, such as a correct, incorrect or partial attempt, confusion about content or procedure, a hint, or a direct answer. The coded sequence is then mapped to a Krashen-inspired state: Under-, Optimally or Over-Challenged.

Applied to the Teacher-Student Chatroom Corpus, the states agree well with human judgement. Accumulated state histories modestly predict success on the next exercise through a mixed-effects model. When these predictions are rolled up into a session-level improvement proxy, sessions with more over-challenging tasks show smaller gains, and optimal challenge beats under-challenge.

The approach turns a hard latent judgement into simple behavioural labelling that LLMs handle reliably. It could serve as an evaluation metric or a reinforcement-learning reward for tutoring systems. The authors stress that the findings are observational only.

Key points

  • Data: TSCC Version 2, with 260 sessions, 2 teachers and 12 students at CEFR levels B1–C2. Several learners list Ukrainian (or Russian/Ukrainian) as their L1.
  • Filtering: 2,527 exercise sequences were reduced to 1,566 usable exercises. GPT-4.1 did the filtering after passing a precision gate above 0.85, checked against 252 human-judged items.
  • Annotation:
    • Human pilot agreement reached κ = 0.96 on 30 exercises.
    • GPT-5.2 (non-reasoning) had to exceed κ > 0.80 against 157 hand-annotated exercises (1,570 turns) before being used on the full set.
    • State-level agreement with humans was κ = 0.79.
  • Mapping window: Trajectories are capped at 10 turns. 93.6% of annotated exercises reached a stop condition within that limit.
  • Limitations:
    • The next-exercise model (GLMM) has an AUC of only 0.61.
    • The improvement proxy is not an external learning measure.
    • The analysis ignores the order of states within a session.
    • Silent or invisible effort pushes labels toward Under-Challenged.
    • The sample is small.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States434 works · 314,419 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 1Research subjects and languages the work is about.Grammatical error correction & learner language · 156 works

Cite

Cite as Ziqi Shu, Shuman Wang and Michael Hardy. 2026. Measuring Optimal Challenge: Trajectory-Based Difficulty Alignment in Open-Ended Language Tutoring. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.46

@inproceedings{shu2026measuring,
  title = {{Measuring Optimal Challenge: Trajectory-Based Difficulty Alignment in Open-Ended Language Tutoring}},
  author = {Ziqi Shu, Shuman Wang and Michael Hardy},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.46},
  url = {https://doi.org/10.18653/v1/2026.bea-1.46}
}

References

18 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Elizabeth L. Bjork and Robert A. Bjork. 2011. Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. In Morton Ann Gernsbacher, Richard W. Pew, Leaetta M. Hough, and James R. Pomerantz, editors, Psychology and the Real World: Essays Illustrating Funda… find
  2. Thiago Brant, Julien Kühn, and Jun Pang. 2026. Estimating exam item difficulty with llms: A benchmark on brazil’s enem corpus. Preprint, arXiv:2602.06631. — [arXiv:2602.06631](https://arxiv.org/abs/2602.06631) find
  3. Norman E. Breslow and David G. Clayton. 1993. Approximate inference in generalized linear mixed models. Journal of the American Statistical Association, 88(421):9–25. find
  4. Andrew Caines, Helen Yannakoudakis, Helena Edmondson, Helen Allen, Pascual Pérez-Paredes, Bill Byrne, and Paula Buttery. 2020. The teacher-student chatroom corpus. Preprint, arXiv:2011.07109. find
  5. Albert T. Corbett and John R. Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4(4):253–278. find
  6. Council of Europe. 2001. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge University Press. find
  7. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120.
  8. Ronald K. Hambleton, Hariharan Swaminathan, and H. Jane Rogers. 1991. Fundamentals of Item Response Theory. Sage Publications, Newbury Park, CA. find
  9. Manu Kapur. 2008. Productive failure. Cognition and Instruction, 26(3):379–424. find
  10. Stephen Krashen. 1977. Some issues relating to the monitor model. In H. Douglas Brown, Carlos Alfredo Yorio, and Ruth H. Crymes, editors, Teaching and Learning English as a Second Language: Trends in Research and Practice: On TESOL ’77: Selected Papers from the Eleventh Annual Convention of Teachers… find
  11. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159. find
  12. Yunting Liu, Shreya Bhandari, and Zachary A. Pardos. 2025. Leveraging llm respondents for item evaluation: A psychometric analysis. British Journal of Educational Technology, 56(3):1028–1052. find
  13. Michael H. Long. 1996. The role of the linguistic environment in second language acquisition. In William C. Ritchie and Tej K. Bhatia, editors, Handbook of Second Language Acquisition, pages 413– 468. Academic Press, San Diego. find
  14. Seyed Parsa Neshaei, Richard Lee Davis, Adam Hazimeh, Bojan Lazarevski, Pierre Dillenbourg, and Tanja Käser. 2024. Towards modeling learner performance with large language models. Preprint, arXiv:2403.14661. — [arXiv:2403.14661](https://arxiv.org/abs/2403.14661) find
  15. Chris Piech, Jonathan Spencer, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge tracing. Preprint, arXiv:1506.05908. find
  16. Petter Törnberg. 2023. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. Preprint, arXiv:2304.06588.
  17. Lev S. Vygotsky. 1978. Mind in Society: The Development of Higher Psychological Processes. Harvard University Press, Cambridge, MA. find
  18. Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is ChatGPT a good NLG evaluator? a preliminary study. In Proceedings of the 4th New Frontiers in Summarization Workshop, pages 1–11, Singapore. Association for Computational Lingu…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shu-2026-measuring-optimal-challenge-trajectory-based/