Elamur Research⌕
☰
home / papers / 2019 / ChID: A Large-scale Chinese IDiom Dataset for Cloze Test

ChID: A Large-scale Chinese IDiom Dataset for Cloze Test

Paper · 2019 · Chujie Zheng, Minlie Huang and Aixin Sun

Summary

Chinese cloze-style reading comprehension has had few benchmarks, and none of them target chengyu (成语). These four-character idioms are often non-compositional and metaphorical, and many have near-synonyms that differ only subtly in usage. To fill this gap, the authors build ChID, a large corpus drawn from news, novels and essays. In each passage, idioms are masked and must be recovered from a list of seven candidates. The authors design candidate lists with embedding-based similarity, which lets them control difficulty by mixing near-synonymous distractors with unrelated ones. They benchmark three cloze models (LM, AR, SAR) and compare ways of representing idioms. The best system, AR, still trails human readers by roughly 15–23 accuracy points. The gap is largest on out-of-domain passages, which contain more rare idioms. Treating each idiom as a single pretrained token works clearly better than building its meaning from its characters. ChID therefore offers a demanding benchmark for figurative-language understanding in Chinese machine reading.

Key points

  • Dataset size and coverage: ChID contains 581K passages and 729K blanks, covering 3,848 idioms from three domains (news, novels, essays).
  • Candidate design: In the main setting, each blank offers the gold idiom, three similar idioms and three random ones. The extra test sets Ran (only dissimilar distractors) and Sim (distractors from the top-10 similar idioms) show that similar distractors make the task harder for both humans and models.
  • Near-synonym study: Annotators examined 200 sampled idioms (Fleiss' kappa 0.479). About 89.5% had at least one near-synonym, and 23% had four or more.
  • Human vs. model accuracy: AR, the best model, scores 72.4 on Test and 62.9 on Out. Humans score 87.1 and 86.2 on the same sets, a gap of 14.6 to 23.3 points.
  • Idiom representation: Using a whole-idiom embedding beats both averaging character embeddings and feeding the characters through an MLP, across all models. For example, AR on Test scores 72.4 with idiom embeddings, 66.5 with the MLP and 63.8 with averaged characters (p < 0.01).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🎓Nanyang Technological UniversitySingapore · Singapore73 works · 24,390 citations🔬Beijing National Research Center for Information Science and TechnologyBeijing · China21 works · 5,843 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksDevelopmental and Educational Psychology · 260 works

🧭 Topics 1Research subjects and languages the work is about.Idioms & multiword expressions · 155 works

Cite

Cite as Chujie Zheng, Minlie Huang and Aixin Sun. 2019. ChID: A Large-scale Chinese IDiom Dataset for Cloze Test. arXiv:1906.01265.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1906.01265 · arXiv 1906.01265

@misc{zheng2019chid,
  title = {{ChID: A Large-scale Chinese IDiom Dataset for Cloze Test}},
  author = {Chujie Zheng, Minlie Huang and Aixin Sun},
  year = {2019},
  doi = {10.48550/arxiv.1906.01265},
  eprint = {1906.01265},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1906.01265}
}

References

39 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16)… find
  2. Dimitra Anastasiou. 2010. Idiom treatment experiments in machine translation. Cambridge Scholars Publishing. find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations.
  4. Christopher M Bishop et al. 1995. Neural networks for pattern recognition. Oxford university press. find
  5. Cristina Cacciari and Patrizia Tabossi. 2014. Idioms: Processing, structure, and interpretation. Psychology Press. find
  6. Fabienne Cap, Manju Nirmal, Marion Weller, and Sabine Schulte Im Walde. 2015. How to account for idiomatic german support verb constructions in statistical machine translation. In Proceedings of the 11th Workshop on Multiword Expressions, pages 19–28. find
  7. Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 2358–2367.
  8. Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017. Attention-overattention neural networks for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 593–602. find
  9. Yiming Cui, Ting Liu, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2018. Dataset for the 7 Jon Jonz. 1991. Cloze item types and second language Our dataset is available at https://github.com/chujiezheng/ChID-Dataset. comprehension. Language testing, 8(1):1–22. first evaluation on chinese ma… ↗
  10. Yiming Cui, Ting Liu, Zhipeng Chen, Shijin Wang, and Guoping Hu. 2016. Consensus attention-based neural networks for chinese reading comprehension. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1777–1786. find
  11. Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017. Gated-attention readers for text comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1832–1846. find
  12. Terrence L Fine. 1999. Feedforward Neural Network Methodology. Springer Science & Business Media. find
  13. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  14. Sandra S Fotos. 1991. The cloze test as an integrative measure of efl proficiency: A substitute for essays on college entrance examinations? Language learning, 41(3):313–336. find
  15. Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, et al. 2018. Dureader: a chinese machine reading comprehension dataset from real-world applications. In Proceedings of the Workshop on Machine Reading for Question Answering, pages 37–46.
  16. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693– 1701.
  17. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks principle: Reading children’s books with explicit memory representations. In International Conference on Learning Representations.
  18. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  19. Ray Jackendoff and Ray S Jackendoff. 2002. Foundations of language: Brain, meaning, grammar, evolution. Oxford University Press, USA. find
  20. Zhiying Jiang, Boliang Zhang, Lifu Huang, and Heng Ji. 2018. Chengyu cloze test. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 154–158. find
  21. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations. find
  22. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785– 794.
  23. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In International Conference on Learning Representations Workshop.
  24. Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017. find
  25. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46–51. find
  26. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268.
  27. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2230– 2235.
  28. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. The lambada dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association f…
  29. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392.
  30. Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. Coqa: A conversational question answering challenge. arXiv preprint arXiv:1808.07042.
  31. Giancarlo D. Salton, Robert J. Ross, and John D. Kelleher. 2014. Evaluation of a substitution method for idiom transformation in statistical machine translation. In MWE@EACL. find
  32. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In International Conference on Learning Representations.
  33. Yutong Shao, Rico Sennrich, Bonnie L. Webber, and Federico Fancellu. 2017. Evaluating machine translation performance on chinese idioms with a blacklist method. In Language Resources and Evaluation Conference. find
  34. Yan Song, Shuming Shi, Jing Li, and Haisong Zhang. 2018. Directional skip-gram: Explicitly distinguishing left and right context for word embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol…
  35. M Sun, J Li, Z Guo, Z Yu, Y Zheng, X Si, and Z Liu. 2016. Thuctc: an efficient chinese text classifier. GitHub Repository. find
  36. Wilson L Taylor. 1953. “cloze procedure”: A new tool for measuring readability. Journalism Bulletin, 30(4):415–433. find
  37. Annie Tremblay. 2011. Proficiency assessment standards in second language acquisition research:“clozing” the gap. Studies in Second Language Acquisition, 33(3):339–372. find
  38. Alison Wray. 2002. Formulaic language and the lexicon. find
  39. Qizhe Xie, Guokun Lai, Zihang Dai, and Eduard Hovy. 2018. Large-scale cloze test dataset created by teachers. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2344–2356. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2019-chid-large-scale-chinese/