Elamur Research
☰
home / papers / 2019 / Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning

Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning

What it is about

Quoref has over 24K span-answer questions on 4.7K Wikipedia paragraphs (about 40% film plot summaries, which mention many characters), written by crowd workers so that answering needs linking two mentions of one entity. To keep out questions answerable by surface cues, a BERT model trained on SQuAD ran in the background and a question was accepted only if that model got it wrong. In a sample of 100, 78% truly need coreference: pronominal 69%, nominal 54%, both 32%, commonsense 10%; about 10% of answers are several spans. Questions often name the target by a description instead of a name («the person who is defied by her daughter» → Hippolyta, from «Hippolyta was granted a daughter, Diana … Diana defies her mother»). The best model reached 70.5 F1 against 93.4 for humans.

Ideas

For Mova

languages/en/35-reference-and-coreference.md §10 (the resolver inside the frame reader), docs/mmm-diagnosis.md (the diagnosis steps). Code: world/src/knowledge.rs (diagnose split), prag/src/frames.rs (Ctx, relative clauses in questions), world/src/ans.rs. Measures: course test counts on pronoun-dependent and deep questions; Quoref dev only as an outside test set, given the Wikipedia licence.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/1908.05803

Summary

Existing reading comprehension benchmarks rarely test whether a model can follow the same entity across a paragraph. This leaves coreference resolution largely unevaluated. The authors address this gap with QUOREF, a crowdsourced English dataset of span-selection questions. The questions were written on Mechanical Turk over Wikipedia paragraphs about films, art and architecture, geography, history and music. Annotators wrote questions whose answers depend on linking two or more mentions of an entity.

Crowdsourced questions often contain surface clues that let models skip the intended reasoning. To prevent this, an uncased base BERT model trained on SQuAD 1.1 ran during annotation as an adversary, and a question was accepted only if this model answered it wrongly. In a manual check of 100 samples, roughly four in five questions could not be solved without coreference.

The strongest tested system, XLNet QA, reaches 70.5 F1 on the test set, while humans score 93.4. This large gap makes QUOREF a useful benchmark for coreference-aware reading comprehension.

Key points

  • Size and splits: over 24K questions on about 4.7K English Wikipedia paragraphs.
    • Train: 19,399 questions. Dev: 2,418. Test: 2,537.
    • Roughly 10% of answers consist of multiple spans.
    • Movie plot summaries make up about 40% of the paragraphs.
  • Manual analysis of 100 questions:
    • 78% need coreference resolution.
    • 69% involve pronominal mentions and 54% involve nominal mentions; 32% need both.
    • 10% call for commonsense reasoning.
  • Model results (test F1):
    • XLNet QA: 70.51
    • BERT QA: 66.39
    • QANet+BERT: 47.20
    • QANet: 38.90
    • Human estimate: 93.41
  • Passage-only baselines: versions of BERT QA and XLNet QA that never see the question score only about 28–29 F1. They mostly pick the most frequent entity in the passage.
  • Error analysis on dev: models partly exploit entity prominence. Among questions BERT QA answered correctly, 51% had an answer that was the first or most frequent entity in the paragraph. Among its wrong answers, the figure was only 12%.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 1908.05803

@misc{dasigi2019quoref,
  title = {{Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning}},
  author = {Dasigi, Pradeep and Liu, Nelson F. and Marasovic, Ana and Smith, Noah A. and Gardner, Matt},
  year = {2019},
  eprint = {1908.05803},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.05803}
}

References

28 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rahul Aralikatte and Anders Søgaard. 2019. Model-based annotation of coreference. arXiv:1906.10724. ↗
  2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. MS MARCO: A human generated machine reading comprehension dataset. In NeurIPS. find our card
  3. Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A large-scale dataset in preschool vocabulary for coreference resolution. In Proc. of EMNLP. find our card
  4. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In Proc. of ACL. find our card
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL. find
  6. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL. find our card
  7. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proc. of Workshop for NLP Open Source Software. find our card
  8. Abbas Ghaddar and Phillippe Langlais. 2016. Wiki-Coref: An English coreference-annotated corpus of Wikipedia articles. In Proc. of LREC. find our card
  9. Anupam Guha, Mohit Iyyer, Danny Bouman, and Jordan Boyd-Graber. 2015. Removing the training wheels: A coreference dataset that entertains humans and challenges computers. In Proc. of NAACL. find
  10. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proc. of NAACL. find
  11. Mandar S. Joshi, Eunsol Choi, Daniel S. Weld, and Luke S. Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proc. of ACL. find our card
  12. Divyansh Kaushik and Zachary Chase Lipton. 2018. How much reading does reading comprehension require? A critical investigation of popular benchmarks. In Proc. of EMNLP. find our card
  13. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: A benchmark for question answering research. In TACL. find our card
  14. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. arXiv:1906.08237. ↗ our card
  15. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proc. of EMNLP. find our card
  16. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. QANet: Combining local convolution with global self-attention for reading comprehension. In Proc. of ICLR. find our card
  17. Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019. Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proc. of NAACL. find
  18. Courtney Napoles, Matthew R. Gormley, and Benjamin Van Durme. 2012. Annotated Gigaword. Technical report. find
  19. Massimo Poesio, Jon Chamberlain, Silviu Paun, Juntao Yu, Alexandra Uma, and Udo Kruschwitz. 2019. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proc. of NAACL. find our card
  20. Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolution with the ARRAU corpus. In Proc. of CRAC Workshop. find
  21. Sameer S. Pradhan, Lance Ramshaw, Ralph Weischedel, Jessica MacBride, and Linnea Micciulla. 2007. Unrestricted coreference: Identifying entities and events in OntoNotes. In Proc. of ICSC. find
  22. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. of EMNLP. find our card
  23. Marta Recasens, Eduard Hovy, and M Antònia Martı́. 2011. Identity, non-identity, and near-identity: Addressing the complexity of coreference. Lingua, 121(6):1138–1152. find
  24. Matthew Richardson, Christopher J. C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proc. of EMNLP. find our card
  25. Mark D. Smucker, Charles L. A. Clarke, and Gordon V. Cormack. 2009. Experiments with clueweb09: Relevance feedback and web tracks. In Proc. of TREC. find
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NeurIPS. find
  27. Yannick Versley. 2008. Vagueness and referential ambiguity in a large-scale annotated corpus. Research on Language and Computation, 6(3):333–353. ↗
  28. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. Proc. of ICCV. find our card