Elamur Research⌕
☰
home / papers / 2023 / ClozEx: A Task toward Generation of English Cloze Explanation

ClozEx: A Task toward Generation of English Cloze Explanation

Paper · 2023 · Zizheng Zhang, Masato Mita and Mamoru Komachi

Summary

The paper tackles automatic explanation of English cloze items for ESL learners: given a sentence with a gap and (usually four) answer options, a system should produce an explanation that reads well (fluency) and actually gives the knowledge needed to choose correctly (validity). No dataset existed for this, and expert-written explanations are expensive, so the authors built one semi-automatically. They took expert-designed questions and explanations, abstracted them into grammar-item patterns, matched those patterns against sentences from a news corpus to create new items and distractors, filled type-specific explanation templates, and had an LLM paraphrase the templates for variety. The result, ClozEx, has over 140k question–explanation pairs covering affix, verb-tense and preposition items. Fine-tuned encoder-decoder models produced the best explanations. Zero-shot LLMs wrote fluent text that often failed to justify the answer. A study comparing metrics with human judgments found that an LLM can reasonably score fluency but not validity, which matters for anyone evaluating explanation generation.

Key points

  • The task takes a gapped sentence plus an option list as input and outputs a free-text explanation, judged on fluency and validity. Explanations are capped at 128 words to limit irrelevant content.
  • The dataset holds more than 140k instances across three TOEIC-prominent question types: affix (POS through prefixes and suffixes), verb-tense, and preposition (sense in context). It is released at github.com/zz-zhang/ClozEx.
  • Baselines include fine-tuned encoder-decoder and decoder-only models, plus zero-shot LLMs given simple prompts. The encoder-decoder models were the most promising, and LLM outputs were fluent but frequently uninformative.
  • Correlations with human judgments (Pearson, per annotator):
    • GPT3.5-turbo fluency scores track manual fluency at roughly 0.57–0.61.
    • GPT-Validity barely correlates with manual validity (about −0.03 to 0.05) and tends to overrate GPT3.5-turbo's own explanations.
    • BLEU correlates weakly overall but noticeably better (around 0.4) once LLM-generated explanations are excluded.
  • The authors list three limitations: no learner study was run to show proficiency gains, only three question types are covered, and the evaluators rely on a closed LLM.

Interesting ideas

  • I1. Instead of hand-writing many items, the authors abstract a few expert examples into patterns tied to grammar items, then instantiate those patterns over a large corpus. This yields scale while inheriting expert quality.
  • I2. Explanations come first from deterministic templates filled with question and pattern facts. An LLM is used only to paraphrase, so the content stays controlled while the wording varies.
  • I3. Distractors are chosen according to the specific language aspect each item tests, such as wrong POS for affix items or wrong tense for verb items. Wrong options therefore probe one skill at a time.
  • I4. The paper treats fluency and validity as separate axes and checks each automatic metric against human scores. An LLM judge turned out reliable for fluency and useless for validity.
  • I5. A reference-based score can unfairly penalize a correct answer phrased in a different style. In this paper, a high BLEU score signalled quality, while a low one did not reliably signal a poor explanation.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Mova could abstract a handful of verified tree→frame analyses into patterns, match them across a corpus, and use the matches as silver training data with known provenance.
  • I2 (back): Generation from the meaning graph could render facts through templates and allow an LLM only a surface-paraphrase role, then verify that the paraphrase still maps back to the same graph.
  • I3 (answer): Self-built test items for the school-course readers could use aspect-targeted distractors, so a failure pinpoints which layer (morphology, tense, preposition sense) broke.
  • I4 (learn): When LLM teacher tiers grade Mova's outputs, their verdicts should be trusted only on style-like axes, and correctness should be checked against gold data or negative controls.
  • I5 (answer): In QA evaluation, overlap with a reference should count as a positive signal only, and non-matching answers should be checked semantically through the frames rather than marked wrong.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tokyo Metropolitan UniversityTokyo · Japan6 works · 337 citations🎓Hitotsubashi UniversityTokyo · Japan6 works · 28 citations

Also on the byline: CyberAgent.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Zizheng Zhang, Masato Mita and Mamoru Komachi. 2023. ClozEx: A Task toward Generation of English Cloze Explanation. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.347

@inproceedings{zhang2023clozex,
  title = {{ClozEx: A Task toward Generation of English Cloze Explanation}},
  author = {Zizheng Zhang, Masato Mita and Mamoru Komachi},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.347},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.347}
}

References

36 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J Charles Alderson. 1979. The cloze procedure and proficiency in English as a foreign language. TESOL quarterly, pages 219–227. ↗
  2. ALTE. 2011. Manual for Language Test Development and Examining: For Use with the CEFR. Language Policy division, Council of Europe. find
  3. Lyle F Bachman. 1985. Performance on cloze tests with fixed-ratio and rational deletions. TESOL Quarterly, 19(3):535–556. ↗
  4. Martin Chodorow, Michael Gamon, and Joel Tetreault. 2010. The utility of article and preposition error correction systems for English language learners: Feedback and assessment. Language Testing, 27(3):419– 436. ↗
  5. Laura Collins. 2007. L1 differences and L2 similarities: teaching verb tenses in English. ELT Journal, 61(4):295–303. ↗
  6. Rui Correia, Jorge Baptista, Maxine Eskenazi, and Nuno Mamede. 2012. Automatic generation of cloze question stems. In International Conference on Computational Processing of the Portuguese Language, pages 168–178. Springer. find
  7. Jennifer Cotter. 2012. Understanding the relationship between reading fluency and reading comprehension: Fluency strategies as a focus for instruction. find
  8. Charles B. Cross. 1991. Explanation and the theory of questions. Erkenntnis (1975-), 34(2):237–260. ↗
  9. Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074–1084, Florence, Italy. Association for Computational Linguistics.
  10. Eileen W Glisan and Victor Drescher. 1993. Textbook grammar: Does it reflect native speaker speech? The Modern Language Journal, 77(1):23–33. ↗
  11. Masamichi Mochizuki and Kazumi Aizawa. 2000. An affix acquisition order for EFL learners: an exploratory study. System, 28(2):291–304. ↗
  12. Hongyu Gong, Jiaqi Mu, Suma Bhat, and Pramod Viswanath. 2018. Preposition sense disambiguation and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1510–1521, Brussels, Belgium. Association for Computational Linguistics. find
  13. Ryo Nagata. 2019. Toward a task of feedback comment generation for writing learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3206–3215, Hong Kong, China. Association for Computational Linguistics. find
  14. Takuya Goto, Tomoko Kojiri, Toyohide Watanabe, Tomoharu Iwata, and Takeshi Yamada. 2010. Automatic generation system of multiple-choice cloze questions and its evaluation. Knowledge Management & E-Learning: An International Journal, 2(3):210–224. ↗
  15. Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. 2017. news-please: A generic news crawler and extractor. In Proceedings of the 15th International Symposium of Information Science, pages 218–223. ↗
  16. Jennifer Hill and Rahul Simha. 2016. Automatic generation of context-based fill-in-the-blank exercises using co-occurrence likelihoods and Google n-grams. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 23–30. find
  17. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438. ↗
  18. Tex Kato. 2017. TOEIC L&R tesuto bunpo mondai deru 1000mon (TOEIC L&R Test Grammar Problems 1000 Questions). ASK Publishing. find
  19. Christine Klein-Braley. 1997. C-tests in the context of reduced redundancy testing: An appraisal. Language testing, 14(1):47–84. ↗
  20. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199–22213. Curran Associates, Inc.
  21. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. ↗
  22. Zhengliang Liu, Xiaowei Yu, Lu Zhang, Zihao Wu, Chao Cao, Haixing Dai, Lin Zhao, Wei Liu, Dinggang Shen, Quanzheng Li, Tianming Liu, Dajiang Zhu, and Xiang Li. 2023. Deid-gpt: Zero-shot medical text de-identification by gpt-4. ↗
  23. Subhadarshi Panda, Frank Palma Gomez, Michael Flor, and Alla Rozovskaya. 2022. Automatic generation of distractors for fill-in-the-blank exercises with round-trip neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 391–401, Dublin, Ireland. Association for Computational Linguistics. find
  24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗
  25. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  26. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. ↗
  27. Patricia M Raymond. 1988. Close procedure in the teaching of reading. TESL Canada journal, pages 91–97. ↗
  28. James Rye. 1982. Cloze procedure and the teaching of reading. London. find
  29. Keisuke Sakaguchi, Yuki Arase, and Mamoru Komachi. 2013. Discriminative approach to fill-in-the-blank quiz generation for language learners. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 238–242, Sofia, Bulgaria. Association for Computational Linguistics. find
  30. Vivek Srikumar and Dan Roth. 2013. Modeling semantic relations expressed by prepositions. Transactions of the Association for Computational Linguistics, 1:231–242. ↗
  31. Wilson L Taylor. 1953. “Cloze procedure”: A new tool for measuring readability. Journalism quarterly, 30(4):415–433. find
  32. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. ↗
  33. Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. Is ChatGPT a good NLG evaluator? a preliminary study. ↗
  34. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023b. Large language models are not fair evaluators. ↗
  35. Joseph Jay Williams, Tania Lombrozo, and Bob Rehder. 2010. Why does explaining help learning? insight from an explanation impairment effect. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 32. find
  36. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2023-clozex-task-generation-english-cloze/