Reasoning Over Paragraph Effects in Situations

Summary

ROPES is a reading comprehension benchmark from the Allen Institute for AI. It tests whether a system can take a cause–effect relationship described in an expository text and apply it to an unfamiliar scenario. Most existing QA datasets reward locating local facts in a passage, which current models already handle at near-human level.

To build ROPES, the authors scraped paragraphs from science textbooks (CK12, OpenStax) and Wikipedia that contained causal or qualitative cue words, then filtered them by hand. Amazon Mechanical Turk workers wrote a new situation for each chosen passage, plus questions whose span answers require linking the situation to the background. Workers were encouraged to write minimal question pairs to discourage shortcuts. The train, dev and test splits were separated by annotator to limit worker bias.

The resulting 14,322 questions were used to evaluate RoBERTa baselines. The strongest system, RoBERTa LARGE additionally fine-tuned on RACE, reached 61.6% F1 on test, compared with 89.0% for expert humans. Since most questions offer two plausible options, this is barely above chance. The result exposes a clear gap in applying knowledge from reading to new contexts.

Key points

Dataset size: 14,322 questions built on 588 worker-selected background passages, drawn from a pool of over 1,000. Topics are mostly physical science (49%) and life science (45%), with some economics (5%).
Splits are separated by annotator: 7 workers for train (10,924 questions), 2 for dev (1,688) and 2 for test (1,710). Splitting by situation gave inflated scores because of annotator bias.
In a 100-question sample, 71% of questions were effect comparisons and 15% were cause comparisons. Grounding the relation was explicit in 67% of cases, needed common sense in 13%, and required bridging a lexical gap in 20%.
Test results: RoBERTa BASE scored 45.5 F1, RoBERTa LARGE 61.1 F1, and RoBERTa LARGE + RACE 61.6 F1. Humans scored 82.7 EM and 89.0 F1, measured on 400 questions.
Removing the background passage cost RoBERTa LARGE little on test (60.4 F1). This suggests models rely on artifacts rather than truly using the background text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

79 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 70 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-5808 · arXiv 1908.05852

@inproceedings{lin2019reasoning,
  title = {{Reasoning Over Paragraph Effects in Situations}},
  author = {Lin, Kevin and Tafjord, Oyvind and Clark, Peter and Gardner, Matt},
  booktitle = {Proceedings of the 2nd Workshop on Machine Reading for Question Answering},
  year = {2019},
  doi = {10.18653/v1/d19-5808},
  eprint = {1908.05852},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-5808}
}

References

11 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Matthew Richardson, Christopher J. C. Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP. find our card
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP. find
Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. arXiv preprint arXiv:1809.01494. find our card
Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. arXiv preprint arXiv:1606.02858. find our card
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. Lecture Notes in Computer Science, pages 177–190. find
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019. Quarel: A dataset and models for answering questions about qualitative relationships. In AAAI. find
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL. find
Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 189–198. find our card
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL. find our card
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan R. Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In EMNLP. find our card
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. EMNLP. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lin-2019-reasoning-over-paragraph-effects-situations/

📄 Reasoning Over Paragraph Effects in Situations