Elamur Research⌕
☰
home / papers / 2026 / PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback

PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback

Paper · 2026 · Wei Xia, Jin Wu, Haoran Shi et al.

Summary

Automated essay scoring systems usually separate grading from feedback. Neural scorers are hard to interpret, and LLM-generated comments ignore how skilled the writer is. PsyScore connects the two through one latent ability estimate θ drawn from Item Response Theory. A BERT encoder for each trait maps an essay to θ, which is clipped to [−3, 3]. θ then feeds a Graded Partial Credit Model (GPCM) layer trained with negative log-likelihood, using grid-search initialization of discrimination and difficulty parameters. The diagnostic vector, averaged over folds, is mapped to a strategy level inspired by the Zone of Proximal Development (ZPD). The strategies range from explicit correction for weaker writers to more open guidance for stronger ones. They steer a multi-agent "generate-and-fuse" pipeline in which DeepSeek-V3.1 merges the agents' drafts. Feedback is judged by LLM pairwise comparison, simulated student revisions and expert ratings. On ASAP++, scoring accuracy is competitive and stable. The feedback is rated as more actionable and better adapted than that of general LLMs, with the biggest revision gains for weak writers. This moves essay scoring toward formative, ability-aware instruction.

Key points

  • Scoring on ASAP++: mean QWK is 0.727 across prompts, with a coefficient of variation of about 1.67% over initialization settings. The trait-level average is 0.714, and the Overall trait reaches 0.777.
  • Simulated revision gains (PsyScore vs. a No-IRT ablation):
    • low-proficiency writers (θ < −1, n=66): 17.38% vs. 6.07%
    • mid-proficiency writers: 11.39% vs. 0.80%
    • high-proficiency writers: 1.11% vs. 0.09%, a small gain attributed to a ceiling effect
  • Pairwise LLM judgments: PsyScore wins over 90% of comparisons on Actionability against GPT-4o and Llama-3. Against the No-IRT variant, it keeps a win rate above 75% on Adaptivity and Tone.
  • Expert ratings (1–5 Likert scale): PsyScore beats GPT-4o, Llama-4 and Qwen-3 on every dimension (p < .001), for example Adaptivity 4.21 vs. 2.79 for GPT-4o. Inter-rater agreement is substantial, with Fleiss' κ of 0.733 for Specificity and 0.690 for Accuracy.
  • Limitations:
    • the multi-agent pipeline is slow at inference;
    • the scorer needs analytic labels for each trait;
    • the revision study is simulated, not a classroom trial.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksEducation · 203 works

🧭 Topics 1Research subjects and languages the work is about.Grammatical error correction & learner language · 156 works

Cite

Cite as Wei Xia, Jin Wu, Haoran Shi, Xiangyu Wang and Chanjin Zheng. 2026. PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.384

@inproceedings{xia2026psyscore,
  title = {{PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback}},
  author = {Wei Xia, Jin Wu, Haoran Shi, Xiangyu Wang and Chanjin Zheng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.384},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.384}
}

References

58 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amy Adair. 2024. AI-Driven Assessment and Scaffolding for Mathematical Modeling and Explanations During Science Investigations. Ph.D. thesis, Rutgers The State University of New Jersey, School of Graduate Studies. find
  2. Frank B Baker. 2001. The basics of item response theory. ERIC. find
  3. Chen Binbin, Bao Lina, Zhang Rui, Zhang Jingyu, Liu Feng, Wang Shuai, and Li Mingjiang. 2024. A multi-strategy computer-assisted efl writing learning system with deep learning incorporated and its effects on learning: A writing feedback perspective. Journal of Educational Computing Research, 61(8):6… find
  4. Seth Chaiklin and 1 others. 2003. The zone of proximal development in vygotskys analysis of learning and instruction. Vygotskys educational theory in cultural context, 1(2):39–64. find
  5. H. Chen, J. Xu, and B. He. 2014. Automated Essay Scoring by Capturing Relative Writing Quality. The Computer Journal, 57(9):1318–1330. find
  6. Minsoo Cho, Jin-Xia Huang, and Oh-Woog Kwon. 2024. Dual-scale BERT using multi-trait representations for holistic and trait-specific essay grading. ETRI Journal, 46(1):82–95. find
  7. Ian Clark. 2012. Formative assessment: Assessment is for self-regulated learning. Educational psychology review, 24(2):205–249. find
  8. Mădălina Cozma, Andrei Butnaru, and Radu Tudor Ionescu. 2018. Automated essay scoring with string kernels and word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 503–509, Melbourne, Australia. Association for C… find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  10. Heejin Do, Yunsu Kim, and Gary Lee. 2024a. Autoregressive score generation for multi-trait essay scoring. In Findings of the Association for Computational Linguistics: EACL 2024, pages 1659–1666, St. Julian’s, Malta. Association for Computational Linguistics. find
  11. Heejin Do, Sangwon Ryu, and Gary Lee. 2024b. Autoregressive multi-trait essay scoring via reinforcement learning with scoring-aware multiple rewards. find
  12. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16427–16438, Miami, Florida, USA. Association for Computational Linguistics. find
  13. Fei Dong, Yue Zhang, and Jie Yang. 2017. Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 153–162, Vancouver, Canada. Association for Computational Linguistics.
  14. Ahmed M ElMassry, Nazar Zaki, Negmeldin AlSheikh, and Mohammed Mediani. 2025. A systematic review of pretrained models in automated essay scoring. IEEE Access. find
  15. Muhammad Faseeh, Abdul Jaleel, Naeem Iqbal, Anwar Ghani, Akmalbek Abdusalomov, Asif Mehmood, and Young-Im Cho. 2024. Hybrid approach to automated essay scoring: Integrating deep learning embeddings with handcrafted linguistic features for improved accuracy. Mathematics, 12(21):3416. find
  16. Brian Frank, Natalie Simper, and James Kaupp. 2018. Formative feedback and scaffolding for developing complex problem solving and modelling outcomes. European Journal of Engineering Education, 43(4):552–568. find
  17. Jiefu Gong, Xiao Hu, Wei Song, Ruiji Fu, Zhichao Sheng, Bo Zhu, Shijin Wang, and Ting Liu. 2021. Iflyea: A chinese essay assessment system with automated rating, review generation, and recommendation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the… find
  18. John Hattie and Helen Timperley. 2007. The power of feedback. Review of educational research, 77(1):81– 112. find
  19. Yue Huang, C. Palermo, Ruitao Liu, and Yong He. 2025. An early review of generative language models in automated writing evaluation: Advancements, challenges, and future directions for automated essay scoring and feedback generation. Chinese/English Journal of Educational Measurement and Evaluation. find
  20. Angenette C. Imbler, S. Clark, T. Young, and Erika Feinauer. 2022. Teaching second-grade students to write science expository text: Does a holistic or analytic rubric provide more meaningful results? Assessing Writing. find
  21. L. Jacobsen and Kira Elena Weber. 2025. The promises and pitfalls of large language models as feedback providers: A study of prompt engineering and the quality of ai-driven feedback. AI. find
  22. Rui Jiang, Yijia Xue, and Dongmian Zou. 2023. Interpretability-aware industrial anomaly detection using autoencoders. IEEE Access, 11:60490–60500. find
  23. Stephen Krashen. 1982. Principles and practice in second language acquisition. find
  24. Rahul Kumar, Sandeep Mathias, Sriparna Saha, and Pushpak Bhattacharyya. 2022. Many hands make light work: Using essay traits to automatically score essays. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie… find
  25. Sabrina Ludwig, Christian Mayer, Christopher Hansen, Kerstin Eilers, and Steffen Brandt. 2021b. Automated Essay Scoring Using Transformer Models. Psych, 3(4):897–915. find
  26. Pfano Mashau and Jabulani C Nyawo. 2021. The use of an online learning platform: A step towards elearning. South African Journal of Higher Education, 35(2):123–143. find
  27. Vivekanandan Kumar and David Boulanger. 2020. Explainable automated essay scoring: Deep learning really has pedagogical value. In Frontiers in education, volume 5, page 572367. Frontiers Media SA. find
  28. Sandeep Mathias and Pushpak Bhattacharyya. 2018. ASAP++: Enriching the ASAP Automated Essay Grading Dataset with Essay Attribute Scores. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 730–734. European Language Resources Association (E… find
  29. Hannah L., Jang E. E., Shah M., and Gupta V. 2023. Validity arguments for automated essay scoring of young students writing traits. Language Assessment Quarterly, 20(4-5):399–420. find
  30. Haile Misgna, Byung-Won On, Ingyu Lee, and G. Choi. 2024. A survey on deep learning-based automated essay scoring and feedback generation. Artificial Intelligence Review, 58. find
  31. Deming Li and Wei Xing. 2025. A comparative study on sustainable development of online education platforms at home and abroad since the twenty-first century based on big data analysis. Education and Information Technologies, pages 1–22. find
  32. Inderjeet Jayakumar Nair, Jiaye Tan, Xiaotian Su, Anne Gere, Xu Wang, and Lu Wang. 2024. Closing the loop: Learning to generate writing feedback via language model simulated student revisions. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16636– 166… find
  33. Xia Li and Wenjing Pan. 2025. Ceaes: Bidirectional reinforcement learning optimization for consistent and explainable essay assessment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26267–26279. find
  34. Xiaoyu Li, Shaoyang Guo, Jin Wu, and Chanjin Zheng. 2025. An interpretable polytomous cognitive diagnosis framework for predicting examinee performance. Information Processing & Management, 62(1):103913. find
  35. Yuanchao Liu, Jiawei Han, Alexander Sboev, and Ilya Makarov. 2024. GEEF: A neural network model for automatic essay feedback generation by integrating writing skills assessment. Expert Systems with Applications, 245:123043. find
  36. Zhexiong Liu, Diane Litman, Elaine L Wang, Tianwen Li, Mason Gobat, Lindsay Clare Matsumura, and Richard Correnti. 2025. eRevise+RF: A writing evaluation system for assessing student essay revisions and providing formative feedback. In Proceedings of the 2025 Conference of the Nations of the America… find
  37. Sabrina Ludwig, Christian Mayer, Christopher Hansen, Kerstin Eilers, and Steffen Brandt. 2021a. Automated essay scoring using transformer models. Psych, 3(4):897–915. find
  38. Masumi ONO, Hiroyuki YAMANISHI, and Yuko HI-JIKATA. 2019. Holistic and analytic assessments of the toefl ibt® integrated writing task. JLTA Journal, 22:65–88. find
  39. Vishakh Padmakumar and He He. 2024. Does writing with language models reduce content diversity? In International Conference on Representation Learning, volume 2024, pages 642–669. find
  40. P. G. Policar, Martin pendl, Toma Curk, and Blaz Zupan. 2025. Automated assignment grading with large language models: insights from a bioinformatics course. Bioinformatics, 41:i21 – i29. find
  41. Fumiko Samejima. 1969. Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34(S1):1–97. find
  42. Alexander Scarlatos, Nigel Fernandez, Christopher Ormerod, Susan Lottridge, and Andrew Lan. 2025. SMART: Simulated students aligned with item response theory for question difficulty prediction. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 25071–250… find
  43. Nils-Jonathan Schaller, Yuning Ding, Andrea Horbach, Jennifer Meyer, and Thorben Jansen. 2024. Fairness in automated essay scoring: A comparative analysis of algorithms on german learner essays from secondary education. pages 210–221. find
  44. Takumi Shibata and Masaki Uto. 2022. Analytic Automated Essay Scoring Based on Deep Neural Networks Integrating Multidimensional Item Response Theory. find
  45. Jinnie Shin, Qi Guo, and Mark J. Gierl. 2021. Automated essay scoring using deep learning algorithms. pages 37–47. find
  46. Lee S Shulman. 1986. Those who understand: Knowledge growth in teaching. Educational researcher, 15(2):4–14. find
  47. Maja Stahl, Leon Biermann, Andreas Nehring, and Henning Wachsmuth. 2024a. Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Generation. arXiv preprint. ArXiv:2404.15845 [cs]. find
  48. Maja Stahl, Leon Biermann, Andreas Nehring, and Henning Wachsmuth. 2024b. Exploring LLM prompting strategies for joint essay scoring and feedback generation. In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 283–298, Mexico City, Mex… find
  49. Jiong Wang and Jie Liu. 2025. T-MES: Trait-aware mix-of-experts representation learning for multi-trait essay scoring. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1224–1236, Abu Dhabi, UAE. Association for Computational Linguistics. find
  50. David J. Wood, Jérôme Seymour Bruner, and Gail P. Ross. 1976. The role of tutoring in problem solving. Journal of child psychology and psychiatry, and allied disciplines, 17 2:89–100. find
  51. Jin Wu and Chanjin Zheng. 2025. Metacd: A meta learning framework for cognitive diagnosis based on continual learning. Preprint, arXiv:2512.22904. — [arXiv:2512.22904](https://arxiv.org/abs/2512.22904) find
  52. Kaixun Yang, Mladen Raković, Dragan Gašević, and Guanliang Chen. 2025. Does theăprompt-based large language model recognize students’ demographics andăintroduce bias inăessay scoring? In Artificial Intelligence in Education, pages 75–89, Cham. Springer Nature Switzerland. find
  53. John Sweller. 1988. Cognitive load during problem solving: Effects on learning. Cognitive science, 12(2):257–285. find
  54. Kaveh Taghipour and Hwee Tou Ng. 2016. A Neural Approach to Automated Essay Scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1882–1891, Austin, Texas. Association for Computational Linguistics.
  55. Masaki Uto and Maomi Ueno. 2018. Item Response Theory Without Restriction of Equal Interval Scale for Raters Score. In Carolyn Penstein Rosé, Roberto Martínez-Maldonado, H. Ulrich Hoppe, Rose Luckin, Manolis Mavrikis, Kaska Porayska-Pomsta, Bruce McLaren, and Benedict Du Boulay, editors, Artificial… find
  56. Erik Voss. 2025. Comparison of traditional machine learning and neural network approaches for automated scoring of second language english essays. Language Testing, 42:369 – 396. find
  57. Lev S Vygotsky. 1978. Mind in society: The development of higher psychological processes, volume 86. Harvard university press. find
  58. Elaine Lin Wang, Lindsay Clare Matsumura, Richard Correnti, Diane Litman, Haoran Zhang, Emily Howe, Ahmed Magooda, and Rafael Quintana. 2020. erevis(ing): Students revision of text evidence use in an automated writing evaluation system. Assessing Writing, 44:100449. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xia-2026-psyscore-psychometrically-aware-framework-trait/