Elamur Research⌕
☰
home / papers / 2019 / QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions

QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions

Paper · 2019 · Oyvind Tafjord, Matt Gardner, Kevin Lin et al.

Summary

Answering questions that depend on qualitative relationships (for example, more of X leads to less of Y) is hard for QA systems. The only earlier dataset focused on this, QuaRel, encoded its knowledge in a small formal ontology. QuaRTz is the first open-domain alternative, and its background knowledge is plain text. The authors used keyword search to pick 400 general qualitative statements from a large corpus. Crowdworkers then wrote situated two-choice questions grounded in each statement and annotated the properties being compared. A second group of workers checked each question and wrote a "flipped" version with the opposite answer, so that systems cannot succeed by simply matching comparative words. The task is to answer each question using the 400-sentence corpus. The strongest BERT-large model is first trained on RACE, then fine-tuned with retrieved knowledge. It reaches 73.7%, while humans score 95%. This leaves a gap of about 20 points and shows that applying arbitrary textual qualitative knowledge is still an open problem.

Key points

  • Dataset size and splits: QuaRTz has 3864 two-way multiple-choice questions, split 2696/384/784 for train/dev/test, and a background corpus of 400 knowledge sentences. Questions from the same seed sentence stay in the same split.
  • Flipped questions: Half of the questions (1932) are flipped versions. When crowdworkers wrote freely, they flipped the relationship only about 10% of the time, and training on that data alone gave below-random accuracy on flipped questions.
  • Accuracy results:
    • BERT-Sci: 54.6
    • BERT (IR), no RACE pre-fine-tuning: 64.4
    • BERT-PFT with no knowledge: 68.8
    • BERT-PFT (IR): 73.7
    • BERT-PFT with ideal knowledge (IR upper bound): 79.8
    • Humans: 95.0
  • Analysis of 100 sampled questions: 68% use comparative wording that differs from the knowledge sentence, about 35% need commonsense, and 26% compare two explicit situations ("worlds").
  • Annotations and implementation: The property annotations can support semantic-parsing approaches. All models were built with AllenNLP, and the data is available at data.allenai.org/quartz.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States304 works · 106,137 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Oyvind Tafjord, Matt Gardner, Kevin Lin and Peter Clark. 2019. QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions. arXiv:1909.03553.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1909.03553 · arXiv 1909.03553

@misc{tafjord2019quartz,
  title = {{QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions}},
  author = {Oyvind Tafjord, Matt Gardner, Kevin Lin and Peter Clark},
  year = {2019},
  doi = {10.48550/arxiv.1909.03553},
  eprint = {1909.03553},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.03553}
}

References

15 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In EMNLP’13.
  2. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In ACL.
  3. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  4. M. Crouse, C. McFate, and K. Forbus. 2018. Learning to build qualitative scenario models from natural language. In Proc. 31st Int. Workshop on Qualitative Reasoning (QR’18). find
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL.
  6. Kenneth D. Forbus. 1984. Qualitative process theory. Artificial Intelligence, 24:85–168. find
  7. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In NLP OSS Workshop at ACL. (arXiv:1803.07640).
  8. Jayant Krishnamurthy, Pradeep Dasigi, and Matthew Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In EMNLP’17. find
  9. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In EMNLP.
  10. Clifton McFate and Kenneth Forbus. 2016. Scaling up linguistic processing of qualitative processes. In Proc. 4th Ann. Conf. on Advances in Cognitive Systems. find
  11. Clifton James McFate, Kenneth D Forbus, and Thomas R Hinrichs. 2014. Using narrative function to extract qualitative information from natural language texts. In AAAI’14. find
  12. Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019. Improving machine reading comprehension with general reading strategies. In NAACL.
  13. Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019. Quarel: A dataset and models for answering questions about qualitative relationships. In AAAI. find
  14. Daniel S Weld and Johan De Kleer. 2013. Readings in qualitative reasoning about physical systems. Morgan Kaufmann. find
  15. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tafjord-2019-quartz-open-domain-dataset-qualitative/