Elamur Research⌕
☰
home / papers / 2019 / Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension

Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension

Paper · 2019 · Kai Sun, Dian Yu, Dong Yu et al.

Summary

Chinese machine reading comprehension (MRC) has mostly relied on datasets where answers are text spans or are stated directly in the passage. Few resources test whether a system can combine a passage with knowledge it brings from outside. To fill this gap, the authors built C3, the first free-form multiple-choice Chinese MRC dataset. It draws on the HSK and MHK exams for learners of Chinese as a second language. C3 has two parts: C3D, built on dialogues, and C3M, built on more formal mixed-genre texts.

The authors annotated which kinds of prior knowledge each question needs: linguistic, domain-specific, or general world knowledge, with the last split into eight subtypes. They then evaluated rule-based and neural models. The strongest model, BERT-wwm-ext, falls far short of human readers. The gap is largest on questions that need outside knowledge and on questions whose wrong options closely resemble the passage wording. Adding machine-translated English data gives only modest gains.

C3 matters because it is a hard benchmark for knowledge-aware and cross-lingual reading comprehension in Chinese.

Key points

  • C3 contains 13,369 documents and 19,577 questions. Almost half of the problems (48.4%) require understanding a dialogue.
  • 86.8% of questions require both the passage and at least one type of prior knowledge. General world knowledge is needed for 57.3% of questions, more than in earlier Chinese MRC datasets.
  • The best model, BERT-wwm-ext, reaches 68.5% accuracy, while human readers reach 96.0%.
  • The authors define a distractor plausibility score based on edit-distance similarity to the document. When a wrong option looks very similar to the passage, accuracy drops for all models and the need for prior knowledge rises.
  • RACE and DREAM were translated into Chinese with Google Translate, giving 94k extra training instances. Adding them raises dev accuracy only from 67.8% to 72.4%. Multilingual BERT-Base trained on C3 plus the original English data scores 63.4%, below Chinese BERT-Base trained on C3 alone (65.7%).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cornell UniversityIthaca · United States89 works · 58,422 citations🏢TencentShenzhen · China66 works · 17,945 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 242 works

Cite

Cite as Kai Sun, Dian Yu, Dong Yu and Claire Cardie. 2019. Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension. arXiv:1904.09679.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.09679 · arXiv 1904.09679

@misc{sun2019investigating,
  title = {{Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension}},
  author = {Kai Sun, Dian Yu, Dong Yu and Claire Cardie},
  year = {2019},
  doi = {10.48550/arxiv.1904.09679},
  eprint = {1904.09679},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.09679}
}

References

62 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marilyn Adams and Bertram Bruce. 1982. Background knowledge and reading comprehension. Reader meets author: Bridging the gap, 13:2– 25. find
  2. Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2016. Embracing data abundance: Booktest dataset for reading comprehension. arXiv preprint, cs.CL/1610.00956v1. find
  3. Gong Cheng, Weixi Zhu, Ziwei Wang, Jianghui Chen, and Yuzhong Qu. 2016. Taking up the gaokao challenge: An information retrieval approach. In Proceedings of the IJCAI, pages 2479–2485, New York City, NY. find
  4. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of the EMNLP, pages 2174–2184, Brussels, Belgium.
  5. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the NAACL-HLT, pages 2924–2936, Minneapolis, MN.
  6. Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter D Turney, and Daniel Khashabi. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In Proceedings of the AAAI, pages 2580–2586, Phoenix, AZ. find
  7. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019. Pre-training with whole word masking for Chinese BERT. arXiv preprint, cs.CL/1906.08101v1.
  8. Yiming Cui, Ting Liu, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2018a. Dataset for the first evaluation on chinese machine reading comprehension. In Proceedings of the LREC, pages 2721–2725, Miyazaki, Japan. find
  9. Yiming Cui, Ting Liu, Zhipeng Chen, Shijin Wang, and Guoping Hu. 2016. Consensus attention-based neural networks for Chinese reading comprehension. In Proceedings of the COLING, pages 1777–1786, Osaka, Japan. find
  10. Yiming Cui, Ting Liu, Li Xiao, Zhipeng Chen, Wentao Ma, Wanxiang Che, Shijin Wang, and Guoping Hu. 2018b. A span-extraction dataset for Chinese machine reading comprehension. In Proceedings of the EMNLP, pages 5882– 5888, Hong Kong, China.
  11. Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen tau Yih, and Peter Clark. 2018. Tracking state changes in procedural text: A challenge dataset and models for process paragraph comprehension. In Proceedings of the NAACL-HLT, pages 1595–1604, New Orleans, LA. find
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proeedings of the NAACL-HLT, pages 4171–4186, Minneapolis, MN.
  13. Philip Edmonds and Graeme Hirst. 2002. Near-synonymy and lexical choice. Computational linguistics, 28(2):105–144.
  14. Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In Proceedings of the SIGKDD, pages 1156–1165, New York City, NY. find
  15. Song Feng, Jun Seok Kang, Polina Kuznetsova, and Yejin Choi. 2013. Connotation lexicon: A dash of sentiment beneath the surface meaning. In Proceedings of the ACL, pages 1774–1784, Sofia, Bulgaria. find
  16. Ralph Grishman, Lynette Hirschman, and Carol Friedman. 1983. Isolating domain dependencies in natural language interfaces. In Proceedings of the ANLP, pages 46–53, Santa Monica, CA. find
  17. Shangmin Guo, Kang Liu, Shizhu He, Cao Liu, Jun Zhao, and Zhuoyu Wei. 2017a. IJCNLP-2017 Task 5: Multi-choice question answering in examinations. In Proceedings of the IJCNLP 2017, Shared Tasks, pages 34–40, Taipei, Taiwan. find
  18. Shangmin Guo, Xiangrong Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2017b. Which is the effective way for Gaokao: Information retrieval or neural networks? In Proceedings of the EACL, pages 111–120, Valencia, Spain. find
  19. Steffen Leo Hansen. 1994. Reasoning with a domain model. In Proceedings of the NODALIDA, pages 111–121, Stockholm, Sweden. find
  20. Yu Hao, Xien Liu, Ji Wu, and Ping Lv. 2019. Exploiting sentence embedding for medical question answering. In Proceedings of the AAAI, pages 938–945, Honolulu, HI. find
  21. Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. 2017. DuReader: a Chinese machine reading comprehension dataset from real-world applications. In Proceedings of the MRQA, pages 37–46, Melbourne, Austral…
  22. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the NIPS, pages 1693–1701, Montreal, Canada.
  23. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks principle: Reading children’s books with explicit memory representations. In Proceedings of the ICLR, Caribe Hilton, Puerto Rico.
  24. Zixian Huang, Yulin Shen, Xiao Li, Yuang Wei, Gong Cheng, Lin Zhou, Xinyu Dai, and Yuzhong Qu. 2019. GeoSQA: A benchmark for scenario-based question answering in the geography domain at high school level. In Proceedings of the EMNLP-IJCNLP, pages 5865–5870, Hong Kong, China. find
  25. Rei Ikuta, Will Styler, Mariah Hamang, Tim O’Gorman, and Martha Palmer. 2014. Challenges of adding causation to richer event descriptions. In Proceedings of the Second Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pages 12–20, Baltimore, MD. find
  26. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint, cs.CL/1705.03551v2.
  27. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the NAACL-HLT, pages 252–262, New Orleans, LA.
  28. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gáa-bor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association of Computational Linguistics, 6:317–328.
  29. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale reading comprehension dataset from examinations. In Proceedings of the EMNLP, pages 785–794, Copenhagen, Denmark.
  30. Adam Lally, Sugato Bagchi, Michael A Barbo-rak, David W Buchanan, Jennifer Chu-Carroll, David A Ferrucci, Michael R Glass, Aditya Kalyanpur, Erik T Mueller, J William Murdock, Siddharth Patwardhan, and John M Prager. 2017. WatsonPaths: Scenario-based question answering and inference over unstructure… find
  31. Douglas B Lenat, Mayank Prakash, and Mary Shepherd. 1985. CYC: Using common sense knowledge to overcome brittleness and knowledge acquisition bottlenecks. AI magazine, 6(4):65–65. find
  32. Peng Li, Wei Li, Zhengyan He, Xuguang Wang, Ying Cao, Jie Zhou, and Wei Xu. 2016. Dataset and neural recurrent sequence labeling model for open-domain factoid question answering. arXiv preprint, cs.CL/1607.06275v2. find
  33. Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du. 2018. Analogical reasoning on chinese morphological and semantic relations. In Proceedings of the ACL, pages 138– 143, Melbourne, Australia. find
  34. Peter LoBue and Alexander Yates. 2011. Types of common-sense knowledge needed for recognizing textual entailment. In Proceedings of the ACL, pages 329–334, Portland, OR.
  35. George Miller. 1998. WordNet: An electronic lexical database. MIT press. find
  36. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. In Proceedings of the NAACL-HLT, pages 839–849, San Diego, CA. find
  37. Hossein Nassaji. 2006. The relationship between depth of vocabulary knowledge and l2 learners’ lexical inferencing strategy use and success. The Modern Language Journal, 90(3):387–401. find
  38. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. arXiv preprint, cs.CL/1611.09268v3.
  39. Tim O’Gorman, Kristin Wright-Bettner, and Martha Palmer. 2016. Richer event description: Integrating event coreference with temporal, causal and bridging annotation. In Proceedings of the CNS, pages 47–56, Austin, TX.
  40. Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018. SemEval-2018 Task 11: Machine comprehension using commonsense knowledge. In Proceedings of the SemEval, pages 747–757, New Orleans, LA. find
  41. Hoifung Poon, Janara Christensen, Pedro Domingos, Oren Etzioni, Raphael Hoffmann, Chloe Kiddon, Thomas Lin, Xiao Ling, Mausam, Alan Ritter, Stefan Schoenmackers, Stephen Soderland, Dan Weld, Fei Wu, and Congle Zhang. 2010. Machine reading at the University of Washington. In Proceedings of the NAACL-… find
  42. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. In Preprint. find
  43. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proceedings of th ACL, pages 784–789, Melbourne, Australia.
  44. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the EMNLP, pages 2383– 2392, Austin, TX.
  45. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249– 266.
  46. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the EMNLP, pages 193–203, Seattle, WA.
  47. Lenhart Schubert. 2002. Can we derive general world knowledge from texts? In Proceedings of the HLT, pages 94–97, San Diego, CA. find
  48. Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai. 2018. DRCD: a Chinese machine reading comprehension dataset. arXiv preprint, cs.CL/1806.00920v3. find
  49. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019a. DREAM: A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217–231.
  50. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019b. ERNIE: Enhanced representation through knowledge integration. arXiv preprint, cs.CL/1904.09223v1.
  51. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the RepL4NLP, pages 191–200, Vancouver, Canada.
  52. Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. We usually don’t like going to the dentist: Using common sense to detect irony on twitter. Computational Linguistics, 44(4):793– 832.
  53. Shuohang Wang, Mo Yu, Shiyu Chang, and Jing Jiang. 2018. A co-matching model for multi-choice reading comprehension. In Proceedings of the ACL, pages 1–6, Melbourne, Australia. find
  54. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association of Computational Linguistics, 6:287–302.
  55. Morton E Winston, Roger Chaffin, and Douglas Herrmann. 1987. A taxonomy of part-whole relations. Cognitive science, 11(4):417–444. find
  56. Fei Xia. 2000. The part-of-speech tagging guidelines for the Penn Chinese Treebank (3.0). IRCS Technical Reports Series, pages 1–43. find
  57. Qizhe Xie, Guokun Lai, Zihang Dai, and Eduard Hovy. 2018. Large-scale cloze test dataset created by teachers. In Proceedings of the EMNLP, pages 2344–2356, Brussels, Belgium. find
  58. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the EMNLP, pages 2369–2380, Brussels, Belgium.
  59. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018a. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint, cs.CL/1810.12885v1.
  60. Xiao Zhang, Ji Wu, Zhiyang He, Xien Liu, and Ying Su. 2018b. Medical exam question answering with large-scale reading comprehension. In Proceedings of the AAAI, pages 5706– 5713, New Orleans, LA. find
  61. Zhuosheng Zhang and Hai Zhao. 2018. One-shot learning for question-answering in Gaokao history challenge. In Proceedings of the COLING, pages 449–461, Santa Fe, NM. find
  62. Chujie Zheng, Minlie Huang, and Aixin Sun. 2019. ChID: A large-scale Chinese idiom dataset for cloze test. In Proceedings of the ACL, pages 778–787, Florence, Italy.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2019-investigating-prior-knowledge-challenging/