Elamur Research
☰
home / papers / 2018 / DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension

DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-1156.pdf

Summary

Most large reading comprehension (RC) benchmarks use factual, descriptive passages, and their questions often share many words with the text that contains the answer. Simple word matching can therefore score well on them. DuoRC addresses this with a corpus built from paired plot descriptions of the same film, one from Wikipedia and one from IMDb, each written by different authors. One group of crowd workers on Amazon Mechanical Turk wrote questions from one version. A separate group then answered those questions using only the other version, either by extracting or by composing the answer, or by marking the question unanswerable. Because the questions and answers come from different texts, they share little vocabulary. Answering therefore requires paraphrase understanding, commonsense and background knowledge, and reasoning over events across several sentences. The authors define two tasks, SelfRC and ParaphraseRC, and test span-extraction and generative baselines adapted from SQuAD-era models. Performance falls sharply compared with SQuAD, especially on ParaphraseRC. This shows that DuoRC exposes weaknesses current models do not handle, and it can complement existing RC benchmarks.

Key points

  • The corpus contains 186,089 unique QA pairs drawn from 7680 Wikipedia/IMDb movie-plot pairs. IMDb plots average about 926 words and Wikipedia plots about 580, with a word-level Jaccard overlap of only about 26%.
  • There are two settings. In SelfRC, questions are answered from the same plot they were written from. In ParaphraseRC, they are answered from the other plot. About 70% of SelfRC answers and 50% of ParaphraseRC answers are exact spans in the passage.
  • On the full test set, the SpanModel reaches 50.56 F1 on SelfRC. On ParaphraseRC it reaches only 21.53 F1, even after rule-based NLP pruning of the long passage. The GenModel performs worse than the SpanModel in both settings.
  • Training on both subsets and testing on both gives 37.42 F1, compared with roughly 86 F1 for similar models on SQuAD.
  • An oracle step that crops the passage around the answer adds only 1.6 accuracy and 4.3 F1 on the span subset. This indicates that passage length is not the main source of difficulty.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cited by

78 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 73 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1156

@inproceedings{saha2018duorc,
  title = {{DuoRC: Towards Complex Language Understanding with Paraphrased Reading Comprehension}},
  author = {Saha, Amrita and Aralikatte, Rahul and Khapra, Mitesh M. and Sankaranarayanan, Karthik},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1156},
  url = {https://doi.org/10.18653/v1/p18-1156}
}

References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D. Manning. 2014. Modeling biological processes for reading comprehension. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL. http://aclweb.org/anthology/D/D14/D14-1159.pdf. ↗
  2. Yiming Cui. 2017. Cloze explorer. https://github.com/ymcui/Eval-on-NN-of-RC/. ↗
  3. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada. pages 1693– 1701. http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend. ↗ our card
  4. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017. pages 2011–2021. http://aclanthology.info/papers/D17-1214/d17-1214. ↗ our card
  5. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers. pages 1601–1611. https://doi.org/10.18653/v1/P17-1147. ↗ our card
  6. Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. CoRR abs/1506.06726. http://arxiv.org/abs/1506.06726. ↗
  7. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017. RACE: large-scale reading comprehension dataset from examinations. CoRR abs/1704.04683. http://arxiv.org/abs/1704.04683. ↗ our card
  8. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James F. Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. CoRR abs/1604.01696. http://arxiv.org/abs/1604.01696. ↗
  9. Preksha Nema, Mitesh M. Khapra, Anirban Laha, and Balaraman Ravindran. 2017. Diversity driven attention model for query-based abstractive summarization. CoRR abs/1704.08300. http://arxiv.org/abs/1704.08300. ↗
  10. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR abs/1611.09268. http://arxiv.org/abs/1611.09268. ↗ our card
  11. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. arXiv preprint arXiv:1608.05457 . ↗
  12. Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon infused phrase embeddings for named entity resolution. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, CoNLL 2014, Baltimore, Maryland, USA, June 26-27, 2014. pages 78– 86. http://aclweb.org/anthology/W/W14/W14-1609.pdf. ↗
  13. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP). pages 1532–1543. http://www.aclweb.org/anthology/D14-1162. ↗
  14. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016a. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 . ↗ our card
  15. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016b. Squad explorer. https://rajpurkar.github.io/SQuAD-explorer/. ↗
  16. Matthew Richardson, Christopher J. C. Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL. pages 193– 203. http://aclweb.org/anthology/D/D13/D13-1020.pdf. ↗ our card
  17. Tomáš Koˇ ciský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics TBD:TBD. https://TBD. ↗ our card
  18. Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. CoRR abs/1611.01603. http://arxiv.org/abs/1611.01603. ↗ our card
  19. Chuanqi Tan, Furu Wei, Nan Yang, Weifeng Lv, and Ming Zhou. 2017. S-net: From answer extraction to answer generation for machine reading comprehension. CoRR abs/1706.04815. http://arxiv.org/abs/1706.04815. ↗
  20. Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. Movieqa: Understanding stories in movies through question-answering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  21. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Walter Daelemans and Miles Osborne, editors, Proceedings of CoNLL-2003. Edmonton, Canada, pages 142–147. find
  22. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. CoRR abs/1611.09830. http://arxiv.org/abs/1611.09830. ↗ our card
  23. Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Making neural QA as simple as possible but not simpler. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Vancouver, Canada, August 3-4, 2017. pages 271–280. https://doi.org/10.18653/v1/K17-1028. ↗
  24. Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic coattention networks for question answering. CoRR abs/1611.01604. http://arxiv.org/abs/1611.01604. ↗ our card