Elamur Research⌕
☰
home / papers / 2026 / Evaluating Transformer Model Family Representations Through Automated Essay Scoring

Evaluating Transformer Model Family Representations Through Automated Essay Scoring

Paper · 2026 · Akchay Ozten and Rodrigo Wilkens

Summary

The paper asks whether hidden states of decoder-only language models, used as fixed features without prompting or fine-tuning, can support Automated Essay Scoring (AES) better than encoder representations. The authors hold everything constant except the embedding source. They extract final-layer vectors from 15 frozen Hugging Face models: 5 encoders (BERT family) and 10 decoders (Mistral, Qwen3, LLaMA, DeepSeek). They then train regression models on two English datasets. The first is AES2, which scores holistic quality without prompt labels. The second is the CEFR-annotated ELG corpus, which measures proficiency. They also study model size and test hybrid setups that add linguistic features from TextDescriptives.

Decoder embeddings beat encoder embeddings on both datasets when embeddings are used alone. Larger models help only slightly. Adding linguistic features improves results further. Frozen decoder vectors reach a level close to that of a fine-tuned BERT. The study shows that the representational properties of generative models matter for AES, independently of prompt design.

Key points

  • Data:
    • AES2 (Learning Agency Lab AES 2.0 competition) has 17,307 essays scored 1–6 and no prompt labels.
    • ELG CEFR English (Breuker, 2023; distributed via UniversalCEFR on HuggingFace) has 712 essays. Its sublevels such as A1-/A1+ were merged into six levels to reduce class imbalance.
  • Models: On the encoder side, the study uses BERT and DeBERTa in both base and Large sizes plus ModernBERT. On the decoder side, it uses Mistral 7B, Qwen3 (0.6B–14B), Llama 1B/3B, and DeepSeek 1.5B/7B. All remain frozen.
  • Pooling: Encoders use the [CLS] token and decoders use mean pooling over final-layer token states. BERT and DeBERTa were also tested with mean pooling to address the asymmetry between the two setups.
  • Hybrid features: TextDescriptives supplies descriptive statistics, lexical richness, syntactic complexity, dependency distance, and coherence indicators. These features remain complementary to the transformer embeddings.
  • Findings: Decoders consistently beat encoders, gains from larger models are modest, and frozen decoder embeddings are competitive with a fine-tuned BERT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ExeterExeter · United Kingdom12 works · 203 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Akchay Ozten and Rodrigo Wilkens. 2026. Evaluating Transformer Model Family Representations Through Automated Essay Scoring. Proceedings of the Joint Workshop on Readability and Text Simplification (READIxTSAR) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/35ytj3qcrwv4

@inproceedings{ozten2026evaluating,
  title = {{Evaluating Transformer Model Family Representations Through Automated Essay Scoring}},
  author = {Akchay Ozten and Rodrigo Wilkens},
  booktitle = {Proceedings of the Joint Workshop on Readability and Text Simplification (READIxTSAR) @ LREC 2026},
  year = {2026},
  doi = {10.63317/35ytj3qcrwv4},
  url = {https://doi.org/10.63317/35ytj3qcrwv4}
}

References

37 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  2. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72.
  3. Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. find
  4. M. Breuker. 2023. Cefr labelling and assessment services. In G. Rehm, editor, European Language Grid, Cognitive Technologies. Springer, Cham. find
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Scott Crossley, Perpetual Baffour, Jules King, Lauryn Burleigh, Walter Reade, and Maggie Demkin. Learning agency lab-automated essay scoring 2.0, kaggle (2024). URL https://kaggle.com/competitions/learning-agency-lab-automated-essay-scoring-2. ↗
  7. T. Dasgupta, A. Naskar, L. Dey, and R. Saha. 2018. Augmenting textual qualitative features in deep convolution recurrent neural network for automatic essay scoring. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, pages 93–102. find
  8. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. 2018. Bert: pre-training of deep bidirectional transformers for language understanding.
  9. P. W. Foltz, D. Laham, and T. K. Landauer. 1999. The intelligent essay assessor: Applications to educational technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 1(2):939–944. find
  10. L. Hansen, L. R. Olsen, and K. Enevoldsen. 2023. Textdescriptives: A python package for calculating a large variety of metrics from text. find
  11. Y. Attali and J. Burstein. 2006. Automated essay scoring with e-rater[r] v.2. The Journal of Technology, Learning, and Assessment, 4(3). find
  12. P. He, J. Gao, and W. Chen. 2021. Debertav3: improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.
  13. P. He, X. Liu, J. Gao, and W. Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention.
  14. J. M. Imperial, A. Barayan, R. Stodden, R. Wilkens, R. M. Sanchez, L. Gao, and H. T. Madabushi. 2025. Universalcefr: Enabling open multilingual research on language proficiency assessment. arXiv preprint arXiv:2506.01419. find
  15. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. 2023. Mistral 7b. find
  16. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  17. Beata Beigman Klebanov and Nitin Madnani. 2022. Automated essay scoring. Springer Nature. find
  18. P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35.
  19. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019. Roberta: a robustly optimized bert pretraining approach.
  20. Llama Team, AI @ Meta. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  21. E. Mayfield and A. W. Black. 2020. Should you fine-tune bert for automated essay scoring? In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 151–162. find
  22. A. Mizumoto and M. Eguchi. 2023. Exploring the potential of using an ai language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2):100050. find
  23. E. B. Page. 1966. The imminence of... grading essays by computer. The Phi Delta Kappan, 47(5):238–243. find
  24. L. Perelman. 2014. When “the state of the art” is counting words. Assessing Writing, 21:104–111. find
  25. A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever. 2018. Improving language understanding by generative pre-training. find
  26. A. Rogers, O. Kovaleva, and A. Rumshisky. 2020. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842–866.
  27. M. D. Shermis and J. Burstein. 2013. Handbook of Automated Essay Evaluation: Current Applications and New Directions, 1st edition. Routledge. find
  28. M. Stahl, L. Biermann, A. Nehring, and H. Wachsmuth. 2024. Exploring llm prompting strategies for joint essay scoring and feedback generation. arXiv preprint arXiv:2404.15845. find
  29. K. Taghipour and H. T. Ng. 2016. A neural approach to automated essay scoring. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 1882–1891.
  30. M. Uto, Y. Xie, and M. Ueno. 2020. Neural automated essay scoring incorporating handcrafted features. In Proceedings of the 28th international conference on computational linguistics, pages 6077–6088. find
  31. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017. Attention is all you need.
  32. Y. Wang, Z. Wei, Y. Zhou, and X. J. Huang. 2018. Automatic essay scoring incorporating rating schema via reinforcement learning. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 791– 797. find
  33. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, and A. M. Rush. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
  34. K. P. Yancey, G. Laflair, A. Verardi, and J. Burstein. 2023. Rating short l2 essays on the cefr scale with gpt-4. In Proceedings of the 18th workshop on innovative use of NLP for building educational applications (BEA 2023), pages 576–584. find
  35. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le. 2019. Xlnet: generalized autoregressive pretraining for language understanding.
  36. T. Zesch, M. Wojatzki, and D. Scholten-Akoun. 2015. Task-independent features for automated essay grading. In Proceedings of the tenth workshop on innovative use of NLP for building educational applications, pages 224–232. find
  37. P. U. Rodriguez, A. Jafari, and C. M. Ormerod. 2019. Language models and automated essay scoring. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ozten-2026-evaluating-transformer-model-family-representations/