Elamur Research⌕
☰
home / papers / 2026 / Using LLMs for item creation: Validating the potential of automatically generated sentence repetition test items for language assessment

Using LLMs for item creation: Validating the potential of automatically generated sentence repetition test items for language assessment

Paper · 2026 · Sarah Löber, Björn Rudzewitz, Yuan Chu et al.

Summary

Writing items for the Elicited Imitation Test (EIT) is slow and requires expertise. In this sentence-repetition task, test takers hear sentences and reproduce them verbatim, so stimuli must be tightly controlled for length and linguistic complexity. Hand-crafted items also make parallel forms hard to produce, which adds to the lack of standardisation in SLA research. The authors asked whether GPT-4o, used through prompting alone, can produce a usable parallel form of two established, validated English EITs. They compared the LLM-generated test with the two human-written forms on linguistic features (grammatical structures, complexity measures) and on psychometric properties, using data from an online study. The generated items differed somewhat from human-written ones in grammatical structure, but their linguistic complexity was not significantly different. Psychometric differences were minor. The authors conclude that LLM-based Automatic Item Generation is a viable route to comparable EIT forms. This matters because it could make scaling, repeated testing and standardisation feasible, even for test creators without a technical background.

Key points

  • GPT-4o generated a 30-sentence English EIT, intended as a parallel form to two widely used human-written EITs.
  • The participant sample was small (n = 73), so the IRT results are only preliminary, and Differential Item Functioning could not be analysed. Dropouts and missing questionnaire data also occurred.
  • EIT scores correlated only weakly to moderately with a C-Test: ρ = 0.26 and 0.29 for EIT 1 and EIT 3, and ρ = 0.35 for EIT 2. Earlier studies reported much higher correlations (r = 0.69 and 0.50). The authors attribute the gap to the C-Test's emphasis on reading and writing rather than listening and speaking.
  • Grammatical constructs were not explicitly requested in the prompts, yet the generated items largely matched the human forms at the supercategory level.
  • Open questions include other LLMs, languages beyond English, larger item pools, and targeted generation of specific grammatical structures.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TübingenTübingen · Germany75 works · 41,081 citations

Also on the byline: Guangzhou City University of Technology.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Sarah Löber, Björn Rudzewitz, Yuan Chu, Mengyuan He, Shiqin Liu, Yushan Ye and Xiaobin Chen. 2026. Using LLMs for item creation: Validating the potential of automatically generated sentence repetition test items for language assessment. Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bea-1.30

@inproceedings{lober2026using,
  title = {{Using LLMs for item creation: Validating the potential of automatically generated sentence repetition test items for language assessment}},
  author = {Sarah Löber, Björn Rudzewitz, Yuan Chu, Mengyuan He, Shiqin Liu, Yushan Ye and Xiaobin Chen},
  booktitle = {Proceedings of the 21st Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.bea-1.30},
  url = {https://doi.org/10.18653/v1/2026.bea-1.30}
}

References

31 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yigal Attali, Andrew Runge, Geoffrey T LaFlair, Kevin Yancey, Sarah Goodwin, Yena Park, and Alina A Von Davier. 2022. The interactive reading task: Transformer-based automatic item generation. Frontiers in Artificial Intelligence, 5:903077. find
  2. Douglas Bates, Martin Maechler, Ben Bolker, Steven Walker, Rune Haubo Bojesen Christensen, Henrik Singmann, Bin Dai, Gabor Grothendieck, Peter Green, and M Ben Bolker. 2015. Package ‘lme4’. convergence, 12(1):2. find
  3. Pritpal Singh Bhullar, Mahesh Joshi, and Ritesh Chugh. 2024. Chatgpt in higher education-a synthesis of the literature and a future research agenda. Education and Information Technologies, 29(16):21501–21522. find
  4. Harriet Wood Bowden. 2016. Assessing second-language oral proficiency for research: The Spanish elicited imitation task. Studies in Second Language Acquisition, 38(4):647–675. find
  5. Ruhan Circi, Juanita Hicks, and Emmanuel Sikali. 2023. Automatic item generation: foundations and machine learning-based approaches for assessments. In Frontiers in Education, volume 8, page 858273. find
  6. Council of Europe. 2001. Common European framework of reference for languages: Learning, teaching, assessment. Cambridge University Press. find
  7. Larry Davis and John Norris. 2021. Developing an innovative elicited imitation task for efficient english proficiency assessment. ETS Research Report Series, 2021(1):1–30. find
  8. Joshua R De Leeuw. 2015. jspsych: A Javascript library for creating behavioral experiments in a Web browser. Behavior Research Methods, 47(1):1–12. find
  9. Jacob Doughty, Zipiao Wan, Anishka Bompelli, Jubahed Qayum, Taozhi Wang, Juran Zhang, Yujia Zheng, Aidan Doyle, Pragnya Sridhar, Arav Agarwal, and 1 others. 2024. A comparative study of AI-generated (GPT-4) and human-crafted MCQs in programming education. In Proceedings of the 26th Australasian Comp… find
  10. Laila El-Hamamsy, María Zapata-Cáceres, Estefanía Martín Barroso, Francesco Mondada, Jessica Dehler Zufferey, and Barbara Bruno. 2022. The competent computational thinking test: Development and validation of an unplugged computational thinking test for upper primary school. Journal of Educational Co… find
  11. Kathy Minhye Kim, Xiaoyi Liu, Daniel R Isbell, and Xiaobin Chen. 2024. A comparison of lab-and web-based elicited imitation: Insights from explicit-implicit L2 grammar knowledge and L2 proficiency. Studies in Second Language Acquisition, pages 1–22. find
  12. Rosemary Erlam. 2006. Elicited imitation as a measure of L2 implicit knowledge: An empirical validation study. Applied Linguistics, 27(3):464–491. find
  13. Filipe Falcão, Daniela Marques Pereira, Nuno Gonçalves, Andre De Champlain, Patrício Costa, and José Miguel Pêgo. 2023. A suggestive approach for assessing item quality, usability and validity of Automatic Item Generation. Advances in Health Sciences Education, 28(5):1441–1465. find
  14. Mark J Gierl, Hollis Lai, and Simon R Turner. 2012. Using automatic item generation to create multiple-choice test items. Medical Education, 46(8):757– 765. find
  15. Aline Godfroid and Kathy Minhye Kim. 2021. The contribution of implicit-statistical learning aptitude to implicit second-language knowledge. Studies in Second Language Acquisition, 43(3):606—-634. find
  16. Ross Hendrickson, Meghan Aitken, Jeremiah McGhee, and Aaron Johnson. 2010. What makes an item difficult? A syntactic, lexical, and morphological study of elicited imitation test items. In Selected Proceedings of the 2008 Second Language Research Forum, pages 48–56. Cascadilla Proceedings Project Som… find
  17. Yan Huang and Lianzhen He. 2016. Automatic generation of short answer questions for reading comprehension assessment. Natural Language Engineering, 22(3):457–489. find
  18. Daniel R. Isbell, Kathy Minhye Kim, and Xiaobin Chen. 2023. Exploring the potential of automated speech recognition for scoring the Korean Elicited Imitation Test. Research Methods in Applied Linguistics, 2(3):100076. find
  19. Daniel R. Isbell and Young-A Son. 2022. Measurement properties of a standardized elicited imitation test: An integrative data analysis. Studies in Second Language Acquisition, 44(3):859–885. find
  20. Kenji Ishihara, Elizabeth Hiser, and Tae Okada. 2003. Modifying C-test for practical purposes. Doshisha Studies in Language and Culture, 5(4):539–568. find
  21. Euigyum Kim, Seewoo Li, Salah Khalil, and Hyo Jeong Shin. 2025. STAIR-AIG: Optimizing the automated item generation process through human-AI collaboration for critical thinking assessment. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA Mariann… find
  22. Sarah Löber, Björn Rudzewitz, Daniela Verratti Souto, Luisa Ribeiro-Flucht, and Xiaobin Chen. 2024. Developing a Web-Based Intelligent Language Assessment Platform powered by Natural Language Processing Technologies. In Proceedings of the 13th Workshop on Natural Language Processing for Computer Ass… find
  23. Wanjing (Anya) Ma, Michael Flor, and Zuowei Wang. 2025. Automatic generation of inference making questions for reading comprehension assessments. In Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), pages 398–414, Vienna, Austria. Association… find
  24. Michael McGuire and Jenifer Larson-Hall. 2025. Assessing Whisper automatic speech recognition and WER scoring for elicited imitation: Steps toward automation. Research Methods in Applied Linguistics, 4(1):100197. find
  25. Samuel Messick. 1995. Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9):741. find
  26. Meik Michalke, Earl Brown, Alberto Mirisola, Alexandre Brulet, Laura Hauser, and Maintainer Meik Michalke. 2021. Package ‘korpus’. find
  27. Lourdes Ortega, Noriko Iwashita, John M Norris, and Sara Rabie. 2002. An investigation of elicited imitation tasks in crosslinguistic SLA research. In Second Language Research Forum, Toronto, pages 3–6. Paper presentation. find
  28. Anne O’Keeffe and Geraldine Mark. 2017. The English Grammar Profile of learner competence: Methodology and key findings. International Journal of Corpus Linguistics, 22(4):457–489. find
  29. W James Popham. 2000. The mismeasurement of educational quality. School Administrator, 57(11):12– 15. find
  30. Yves Rosseel. 2012. lavaan: An r package for structural equation modeling. Journal of statistical software, 48:1–36. find
  31. Nelly Sagirov and Xiaobin Chen. 2025. POLKE: A system for comprehensively annotating pedagogically-oriented grammatical structure use in language production. Manuscript submitted for publication to Behavior Research Methods. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lober-2026-llms-item-creation-validating-potential/