Elamur Research⌕
☰
home / papers / 2023 / Automated Generation of Multiple-Choice Cloze Questions for Assessing English Vocabulary Using GPT-turbo 3.5

Automated Generation of Multiple-Choice Cloze Questions for Assessing English Vocabulary Using GPT-turbo 3.5

Paper · 2023 · Qiao Wang, Ralph Rose, Naho Orita et al.

Summary

Writing multiple-choice cloze (MCC) vocabulary items by hand is slow for teachers and large programmes. The authors replace their earlier rule- and corpus-based Word Quiz Constructor (WQC) with a GPT-backed generator inside their VocaTT engine. The Python pipeline has three stages. First, LemmInflect expands each target headword into a "word group" of POS-labelled inflected forms. Second, GPT 3.5-turbo writes a short academic sentence around a randomly chosen form with a fixed POS, and that form is then blanked out. Third, same-POS candidates are drawn from other words in the list, and GPT keeps those it judges grammatical but semantically wrong in the slot. The authors built 60 items from the first sublist of the Academic Word List (AWL), and paid expert reviewers judged them. Well-formedness was 75% for stems and 66.85% for distractors, roughly double WQC's rates. Error analysis splits the blame between the inflection tool and the LLM. This suggests that cheap LLM generation still needs validation layers before items can go to learners without human correction.

Key points

  • Data and design choices
    • The data were the 60 headwords of AWL sublist 1, with one question per headword.
    • Stems were capped at 20 words, could not begin with the blank, and had to use the key only once.
    • Each item had three distractors.
  • Tools
    • LemmInflect was chosen over pyInflect because it covered more AWL inflections and GPT could interpret its Penn-style tags.
    • GPT returned JSON and marked the key with backticks.
  • Distractor loop
    • The program takes batches of 10 same-POS candidates from other word groups.
    • A candidate is kept when GPT judges syntax true and semantics false.
    • Three are sampled from the kept candidates, and new rounds run until the pool of that POS runs out.
  • Results against WQC
    • Stems were 75% well-formed, compared with 34.93% for WQC.
    • Distractors were 66.85% well-formed, compared with 38.56% for WQC.
    • The WQC figures come from 1128 earlier AWL items.
  • Error sources and cost
    • Errors were attributed to LemmInflect in 19 cases, to GPT in 21, and to both in 7.
    • GPT sometimes swapped the key, for example writing "indicate" instead of "major", or changed its POS.
    • Generating the 60 items took about 15 minutes, mostly spent waiting on API calls.

Interesting ideas

  • I1. Draw distractors only from words the learner has already studied in the same sublist, so a wrong option can't be ruled out simply because it looks unfamiliar; this removes an unintended clue from the test.
  • I2. A good distractor is defined by two separate judgements: it must fit the sentence grammatically and must fail to fit it in meaning. Splitting the decision into syntax and semantics makes each rejection explainable.
  • I3. Ask the generator for a specific inflected form plus its POS tag, then verify that the output actually kept that form and tag; the paper shows that skipping this check lets the model silently change the key.
  • I4. The authors propose accepting a POS/inflection entry only when two independent sources agree, for example LemmInflect and another tagger or GPT; this cheaply filters out lexicon noise before it spreads into later stages.
  • I5. Judging the full sentence with the candidate inserted is more reliable than judging the word alone, because it exposes problems such as verb transitivity.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): When Mova builds or solves school-course tests, it could restrict foils to vocabulary from the same lesson unit, so that guessing by unfamiliarity is controlled for.
  • I2 (back): Mova could check generated foils with a UD parse for grammatical fit and with frames/roles for semantic misfit, replacing the LLM judge with transparent checks.
  • I3 (back): During generation, Mova could re-tag its own output and reject sentences where the requested lemma, form or POS drifted.
  • I4 (learn): This fits the silver-data scheme of agreement between several witnesses, applied here to morphology lexicon entries rather than whole parses.
  • I5 (tree): Each filled-in candidate sentence could be run through the parser, and low-confidence or ill-formed trees (e.g., a missing obj for a transitive verb) could flag bad options.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Waseda UniversityTokyo · Japan20 works · 584 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 works

Cite

Cite as Qiao Wang, Ralph Rose, Naho Orita and Ayaka Sugawara. 2023. Automated Generation of Multiple-Choice Cloze Questions for Assessing English Vocabulary Using GPT-turbo 3.5. Proceedings of the Joint 3rd International Conference on Natural Language Processing for Digital Humanities and 8th International Workshop on Computational Linguistics for Uralic Languages.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{wang2023automated,
  title = {{Automated Generation of Multiple-Choice Cloze Questions for Assessing English Vocabulary Using GPT-turbo 3.5}},
  author = {Qiao Wang, Ralph Rose, Naho Orita and Ayaka Sugawara},
  booktitle = {Proceedings of the Joint 3rd International Conference on Natural Language Processing for Digital Humanities and 8th International Workshop on Computational Linguistics for Uralic Languages},
  year = {2023},
  url = {https://aclanthology.org/2023.nlp4dh-1.7/}
}

References

18 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Malak Abdullah, Alia Madain, and Yaser Jararweh. 2022. Chatgpt: Fundamentals, applications and social impacts. In 2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS), pages 1–8. IEEE. find
  2. Mofareh Alqahtani et al. 2015. The importance of vocabulary in language learning and how to be taught. International journal of teaching and education, 3(3):21–34. ↗
  3. Jonathan Brown, Gwen Frishkoff, and Maxine Eskenazi. 2005. Automatic question generation for vocabulary assessment. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 819– 826. find
  4. Ralph L. Rose, Naho Orita, Ayaka Sugawara, and Qiao Wang. 2022. Evaluation dataset of multiple-choice cloze items for vocabulary training and testing. In Adjunct Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2022 ACM International Symposium on Wearable Computers, pages 292–298. find
  5. David Coniam. 1997. A computerised english language proofing cloze program. Computer Assisted Language Learning, 10(1):83–97. ↗
  6. Norbert Schmitt. 1997. Vocabulary learning strategies. In Norbert Schmitt and Michael McCarthy, editors, Vocabulary: Description, acquisition and pedagogy, pages 199–227. Cambridge University Press, Cambridge, UK. find
  7. Averil Coxhead. 2000. A new academic word list. TESOL quarterly, 34(2):213–238. ↗
  8. Natasa Gisev, J Simon Bell, and Timothy F Chen. 2013. Interrater agreement and interrater reliability: key concepts, approaches, and applications. Research in Social and Administrative Pharmacy, 9(3):330–338. ↗
  9. Cheryl Boyd Zimmerman. 1997. Do reading and interactive vocabulary instruction make a difference? an empirical study. TESOL quarterly, 31(1):121–140. ↗
  10. Arthur C Graesser. 2001. Question generation as a learning multiplier in distributed learning environments, volume 1121. US Army Research Institute for the Behavioral and Social Sciences. find
  11. Gordon A. Hale, Charles W. Stansfield, Donald A. Rock, Marilyn M. Hicks, Frances A. Butler, and JR John W. Oller. 1989. The relation of multiple-choice cloze items to the test of english as a foreign language. Language Testing, 6(1):47–76. ↗
  12. K. Lee, S. Kweon, H. Kim, and G. Lee. 2013. Filtering-based automatic cloze test generation. In Proceedings of Speech and Language Technology in Education (SLaTE), page 72–76. find
  13. C. Liu, C. Wang, Z. Gao, and S. Huang. 2005. Applications of lexical information for algorithmically composing multiple-choice cloze items. In Proceedings of the 2nd Workshop on Building Educational Applications Using NLP, page 1–8. find
  14. Paul Nation. 2022. Teaching and learning vocabulary. In Eli Hinkel, editor, Handbook of Practical Second Language Teaching and Learning, pages 397–408. Routledge, New York, NY, USA. ↗
  15. Juan Pino, Michael Heilman, and Maxine Eskenazi. 2008. A selection strategy to improve cloze question quality. In Proceedings of the Workshop on Intelligent Tutoring Systems for Ill-Defined Domains. 9th International Conference on Intelligent Tutoring Systems, Montreal, Canada, pages 22–32. find
  16. Ralph L. Rose. 2014. Automated vocabulary quiz creation using online and offline corpora. Poster presentation at Teaching and Language Corpora (TaLC) conference, Lancaster University, UK. find
  17. Ralph L. Rose. 2016. Automatic word quiz construction using regular and simple english wikipedia. In INTED2016 Proceedings, pages 8032–8040. IATED. find
  18. Ralph L. Rose. 2020. Improving the production efficiency and well-formedness of automatically-generated multiple-choice cloze vocabulary questions. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 7094– 7101. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2023-automated-generation-multiple-choice-cloze/