Elamur Research⌕
☰
home / papers / 2025 / End-to-End Automated Item Generation and Scoring for Adaptive English Writing Assessment with Large Language Models

End-to-End Automated Item Generation and Scoring for Adaptive English Writing Assessment with Large Language Models

Paper · 2025 · Kamel Nebhi, Amrita Panesar and Hans Bantilan

Summary

The paper tackles a bottleneck in adaptive English writing tests: producing many new, level-calibrated writing prompts and getting enough labelled responses to score them automatically. Working on the EF Standard English Test (EFSET), the authors build a pipeline with four parts. GPT-4o generates new short essay prompts for a requested topic and proficiency level using few-shot examples. Human experts then filter these prompts. Next, a LLaMA 3.1 (8B) model is fine-tuned with ORPO on real candidate answers whose labels come from a majority vote of three LLM annotators. This model then writes about 200 synthetic, level-varied student responses per new item. Finally, these item–response pairs retrain the existing RoBERTa scorer. Several transformer classifiers are compared on a fixed test set, and RoBERTa-base performs best. The synthetic items also show a spread of difficulty and discrimination under an IRT analysis. The work matters because it shows how to grow an item bank and its scoring model together with little manual authoring or annotation.

Key points

  • Prompt generation uses about 20 hand-curated examples. Each pairs a proficiency level with a prompt of at most 25 words, and GPT-4o adds a topic label to form (prompt, level, topic) triplets.
  • Training labels come from three LLM annotators combined by majority vote. Only samples where the annotators agree strongly are kept.
  • The response simulator is LLaMA 3.1 8B fine-tuned via ORPO. It yields roughly 200 synthetic answers per validated item.
  • On an 800-sample test set balanced across levels and topics, the macro scores were:
    • RoBERTa-base: P 0.83, R 0.81, F1 0.82
    • Flan-T5 Base: F1 0.80
    • BERT-base-uncased: F1 0.78
  • Synthetic items were examined with Item Characteristic Curves from a Two-Parameter Logistic (2PL) IRT model. Representative easy, medium and difficult items differed in both difficulty and discrimination.

Interesting ideas

  • I1. Pooling labels from several independent LLM annotators and discarding low-agreement cases turns cheap, noisy labels into a smaller but more trustworthy training set.
  • I2. Conditioning generation on explicit (level, topic) pairs, with few-shot examples chosen for similarity to the request, gives controllable output instead of generic samples.
  • I3. A small fine-tuned model can imitate learners at different skill levels, producing deliberately imperfect answers that a scorer can learn from.
  • I4. Fitting a 2PL IRT model treats each test item as having its own difficulty and its own power to separate strong from weak takers, not just a pass rate.
  • I5. Framing a classification label as generated text let an instruction-tuned seq2seq model nearly match a dedicated encoder classifier.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): This matches the witness-parser silver data scheme and suggests applying the same agreement threshold to LLM-teacher labels for frames or roles before training on them.
  • I2 (answer): School-course test questions could be generated per grade and subject by giving the generator matched exemplars, which yields graded practice sets for the student readers.
  • I3 (learn): Level-varied imperfect answers could serve as negative controls, checking that Mova's answer checker rejects plausible but wrong student-like responses rather than only accepting correct ones.
  • I4 (answer): Running Mova's evaluation questions through a 2PL fit across model versions could flag items that fail to discriminate, so they can be removed from benchmarks.
  • I5 (learn): When adding new label sets, such as speech-act types, a generative teacher could be tried first as a quick baseline before training a specialised classifier.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Kamel Nebhi, Amrita Panesar and Hans Bantilan. 2025. End-to-End Automated Item Generation and Scoring for Adaptive English Writing Assessment with Large Language Models. Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.bea-1.73

@inproceedings{nebhi2025end,
  title = {{End-to-End Automated Item Generation and Scoring for Adaptive English Writing Assessment with Large Language Models}},
  author = {Kamel Nebhi, Amrita Panesar and Hans Bantilan},
  booktitle = {Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025)},
  year = {2025},
  doi = {10.18653/v1/2025.bea-1.73},
  url = {https://doi.org/10.18653/v1/2025.bea-1.73}
}

References

25 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. Introducing claude 3. https://www.anthropic.com/news/claude-3-family. ↗
  2. Elizabeth Mayfield and Alan W. Black. 2020. Automated scoring of written essays with transformer models. In Proc. BEA at ACL 2020. find
  3. Kamel Nebhi and György Szaszák. 2023. Automatic assessment of spoken english proficiency based on multimodal and multitask transformers. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 769–776. find
  4. Frank B Baker and Seock-Ho Kim. 2004. Item response theory: Parameter estimation techniques. CRC press. find
  5. Isaac I. Bejar and 1 others. 2003. A template-based approach to the generation of test items. In Principles and Practice in Automated Item Generation. find
  6. OpenAI. 2024. Gpt-4o technical report. arXiv preprint arXiv:2405.16408. ↗
  7. Tom B. Brown and 1 others. 2020. Language models are few-shot learners. NeurIPS.
  8. Hugo Touvron, Thibaut Lavril, and 1 others. 2024. Llama 3: Open foundation and instruction models. arXiv preprint arXiv:2404.14219. ↗
  9. Tom V. et al. Brown. 2023. The duolingo english test interactive writing task. In Proc. BEA Workshop at ACL 2023. find
  10. Christopher et al. Clark. 2021. Generating synthetic data for improved accuracy in educational nlp tasks. arXiv preprint arXiv:2106.05071. ↗
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  12. Mark J. Gierl and Thomas M. Haladyna. 2012. Item Generation for Test Development. Routledge. find
  13. Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691. ↗
  14. Zheng-Xin Jiang and 1 others. 2023. Mistral: A simple yet effective baseline for instruction-tuned large language models. arXiv preprint arXiv:2310.06825. ↗
  15. Maryam et al. Kurdi. 2023. Controlled generation of assessment items for educational applications using language models. In Proc. BEA Workshop at ACL 2023. find
  16. LangChain. 2025. Few shot prompt template. https://python.langchain.com/api_reference/ core/prompts/langchain_core.prompts.few_ shot.FewShotPromptTemplate.html. ↗
  17. Yizhong Wang, Zhiwei Zhang, Zexuan Zhong, Zhe Gan, Jingjing Liu, and Noah A. Smith. 2024. Simulating candidates in educational assessment with large language models. arXiv preprint arXiv:2401.07043. ↗
  18. Luke Williams and 1 others. 2022. The duolingo english test: 2022 technical report. In arXiv preprint arXiv:2206.01056. ↗
  19. Xuechen Yao, Ankur P. Parikh, Noah Constant, Dwi Susanti, and Heng Ji. 2024. Leveraging llm-respondents for item evaluation: a psychometric analysis. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1602–1628. find
  20. Xu Zhai, Xiang Wan, Qian Jin, and Eduard Hovy. 2023. Automatic generation of language assessment tasks using large language models. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 340– 350. find
  21. Diyi Zhang and 1 others. 2022. Automatic generation of factual reading comprehension questions with large language models. In Proceedings of ACL 2022. find
  22. Ximing et al. Liu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634. ↗
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  24. Shayne Longpre, Christopher Patil, Abhay Rao, Albert Webson, Le Hou, Pengfei Liu, and 1 others. 2023. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. ↗
  25. Zeyi et al. Mai. 2022. Towards generalizable essay scoring with language models and augmented data. In NAACL 2022. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nebhi-2025-end-end-automated-item-generation/