Elamur Research⌕
☰
home / papers / 2018 / DRCD: a Chinese Machine Reading Comprehension Dataset

DRCD: a Chinese Machine Reading Comprehension Dataset

Paper · 2018 · Chih Chieh Shao, Trois Liu, Yuting Lai et al.

Summary

DRCD (Delta Reading Comprehension Dataset) fills a gap: no large-scale machine reading comprehension (MRC) benchmark existed for traditional Chinese, since most resources are English or simplified Chinese. The authors, from Delta Research Center, chose the span-based format, where each answer is a contiguous fragment of a given passage. They argue this format tests comprehension on its own, without retrieval.

Following the SQuAD collection procedure, they selected high-PageRank Chinese Wikipedia articles. Annotators cleaned these articles, split them into topical paragraphs of 250–1,500 characters, and wrote questions in their own words with answers located in the text. A second annotator answered the dev and test questions to estimate human performance.

Baselines range from a TF-IDF sentence-matching method through R-Net, BiDAF and QANet to fine-tuned Chinese BERT. BERT performed best (F1 89.59%, EM 82.34%) but its F1 still trails humans (93.30%). The dataset is publicly released and intended as a source for transfer learning in traditional Chinese question answering and search.

Key points

  • Size: 10,014 paragraphs from 2,108 Chinese Wikipedia articles, split into train (26,932 questions / 8,014 paragraphs), dev (3,524 / 1,000) and test (3,485 / 1,000).
    • These splits sum to 33,941 QA pairs.
    • The conclusion instead cites 40,410 pairs, a discrepancy the paper does not explain.
  • Lengths: paragraphs average 435.8 characters, questions 21.07 and answers 4.86. Paragraphs are much longer than in SQuAD because annotators split text by topic.
  • Answer types: 70.45% entity, 18.03% numeric and 11.50% description (answers over 6 characters).
  • Question types: "which" (30.96%) and "what" (28.42%) are the most common; "why" accounts for only 0.27%.
  • Test-set F1 / EM:
    • TF-IDF: ~17% / 0.05%
    • R-Net: 38% / 23.8%
    • BiDAF: 51.18% / 28.08%
    • QANet (with jieba tokenization): 78.03% / 65.56%
    • BERT: 89.59% / 82.34%
    • Human: 93.30% / 80.43%

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 487 works

Cite

Cite as Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng and Sam Tsai. 2018. DRCD: a Chinese Machine Reading Comprehension Dataset. arXiv:1806.00920.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1806.00920 · arXiv 1806.00920

@misc{shao2018drcd,
  title = {{DRCD: a Chinese Machine Reading Comprehension Dataset}},
  author = {Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng and Sam Tsai},
  year = {2018},
  doi = {10.48550/arxiv.1806.00920},
  eprint = {1806.00920},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1806.00920}
}

References

11 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693– 1701.
  2. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301.
  3. Yiming Cui, Ting Liu, Zhipeng Chen, Shijin Wang, and Guoping Hu. 2016. Consensus attentionbased neural networks for chinese reading comprehension. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. find
  4. The COLING 2016 Organizing Committee, pages 1777– 1786. http://aclweb.org/anthology/C16-1167. ↗
  5. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683.
  6. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
  7. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge comprehension. ACL. dataset for reading Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms Marco: a Human Generated Mach…
  8. Wei He, Kai Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, et al. 2017. DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications. arXiv preprint arXiv:1711.05073 (2017).
  9. Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated Self-Matching Networks for Reading Comprehension and Question Answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 189–198
  10. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension.
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 (2018).

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shao-2018-drcd-chinese-machine-reading-comprehension/