Elamur Research
☰
home / papers / 2017 / DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications

DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1711.05073

Summary

Existing machine reading comprehension (MRC) benchmarks often rely on artificial setups such as cloze deletion, exam-style multiple choice, or questions written by crowdworkers after reading a passage. They also focus mostly on factual questions answerable by extracting a single span. DuReader, from Baidu, is a large open-domain Chinese MRC dataset built from real user queries in Baidu Search logs. Each question is paired with up to five full documents retrieved from either Baidu Search or the community QA site Baidu Zhidao, and annotators wrote the answers themselves. Every question is labelled as Entity, Description or YesNo, and separately as Fact or Opinion. The authors also introduce an opinion-aware evaluation scheme for yes-no questions. In their experiments, strong baselines such as BiDAF and Match-LSTM fall well short of human performance. The data, baselines and a public leaderboard were released. DuReader matters because it moves MRC research toward realistic, multi-document, opinion-heavy questions where span extraction is insufficient.

Key points

  • Scale: about 200K questions, 420K human-written answers and 1M documents, which the authors describe as the largest Chinese MRC dataset at the time. Questions were filtered from 1M frequent queries using a classifier and then checked by human annotators.
  • A pilot study of 1,000 search queries showed that all six question types are common. Opinion questions made up about 33.8% and YesNo questions about 15.6%, while fact-entity questions were only 23.4%.
  • Full documents are provided instead of single paragraphs, so choosing the relevant paragraph is a key bottleneck for systems.
  • Opinion-aware evaluation: for yes-no questions, systems output an answer plus a Yes/No/Depend label. BLEU and Rouge are then computed only against references that share the same label. Adding a classifier based on Match-LSTM improved scores under this scheme.
  • On ROUGE-L, the BiDAF baseline scored 39.0 and humans 57.4. During the shared task, one competition team reached 51.2.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

118 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 107 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1711.05073 · arXiv 1711.05073

@misc{he2017dureader,
  title = {{DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications}},
  author = {He, Wei and Liu, Kai and Liu, Jing and Lyu, Yajuan and Zhao, Shiqi and Xiao, Xinyan and Liu, Yuan and Wang, Yizhong and Wu, Hua and She, Qiaoqiao and Liu, Xuan and Wu, Tian and Wang, Haifeng},
  year = {2017},
  doi = {10.48550/arxiv.1711.05073},
  eprint = {1711.05073},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1711.05073}
}

References

19 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017. Attention-overattention neural networks for reading comprehension. In Proceedings of 55th Annual Meeting of the Association for Computational Linguistics, pages 593–602. find
  2. Yiming Cui, Ting Liu, Zhipeng Chen, Shijin Wang, and Guoping Hu. 2016. Consensus attention-based neural networks for chinese reading comprehension. find
  3. Matthew Dunn, Levent Sagun, Mike Higgins, Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179. find our card
  4. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693– 1701. find our card
  5. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301. find our card
  6. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. CoRR. find our card
  7. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR. find
  8. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2017. The narrativeqa reading comprehension challenge. arXiv preprint arXiv:1712.07040. find our card
  9. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683. find our card
  10. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74–81. find
  11. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268. find our card
  12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. find our card
  13. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. find our card
  14. Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193–203. find our card
  15. Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. CoRR. find our card
  16. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200. find our card
  17. Shuohang Wang and Jing Jiang. 2017. Machine comprehension using match-lstm and answer pointer. In ICLR, pages 1–15. find our card
  18. Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated Self-Matching Networks for Reading Comprehension and Question Answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 189–198. find
  19. Caiming Xiong, Victor Zhong, and Richard Socher. 2017. Dynamic coattention networks for question answering. In Proceedings of International Conference on Learning Representations. find our card