Elamur Research⌕
☰
home / papers / 2018 / Transforming Question Answering Datasets Into Natural Language Inference Datasets

Transforming Question Answering Datasets Into Natural Language Inference Datasets

Paper · 2018 · Dorottya Demszky, Kelvin Guu and Percy Liang

Summary

Widely used NLI benchmarks such as SNLI and MultiNLI cover phenomena like negation and quantification but rarely require reasoning across several sentences. The authors build NLI data automatically from existing question answering corpora, which are plentiful and span many reasoning types. The central step, QA2D, rewrites a question and an answer as a single declarative statement. That statement becomes the hypothesis and the QA passage serves as the premise. Correct answers produce entailed pairs. Wrong multiple-choice options or unanswerable questions produce non-entailed pairs.

They tried three ways of doing QA2D:

  • a rule-based converter built on dependency parses;
  • crowdsourcing, where workers sometimes edited the rule system's draft;
  • a neural sequence model trained on the roughly 100k collected triples, mostly from SQuAD.

The neural model transfers well to other domains. Applied to five QA corpora, it yields QA-NLI, a free resource of over 500k examples. QA-NLI covers multi-sentence, meta-level and presupposition-based inference, and it seems to avoid some annotation artifacts found in earlier NLI datasets.

Key points

  • Source corpora: MovieQA, NewsQA, QAMR, RACE and SQuAD, covering movie plots, CNN news, Wikipedia/Wikinews and English exams. Passages range from one sentence to multiple paragraphs.
  • Rule-based QA2D: parsing and tagging relied on the Stanford Graph-Based Neural Dependency Parser. Errors from that parser explained about 10% of the rule system's mistakes.
  • Neural QA2D quality: its outputs exactly matched a human reference in 45–57% of cases, with BLEU between 73 and 83 depending on the dataset.
  • Human check on 2,000 MovieQA pairs: over 90% of pairs built from correct answers were judged likely true. Around 80% of pairs from incorrect options were judged unlikely. Labels are binary (entailment vs. non-entailment).
  • Annotation artifacts: a PMI analysis following Gururangan et al. (2018) found that non-entailed hypotheses lack telltale negation words, and hypothesis length barely correlates with the label.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,685 worksQuestion answering · 487 works

Cite

Cite as Dorottya Demszky, Kelvin Guu and Percy Liang. 2018. Transforming Question Answering Datasets Into Natural Language Inference Datasets. arXiv:1809.02922.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1809.02922 · arXiv 1809.02922

@misc{demszky2018transforming,
  title = {{Transforming Question Answering Datasets Into Natural Language Inference Datasets}},
  author = {Dorottya Demszky, Kelvin Guu and Percy Liang},
  year = {2018},
  doi = {10.48550/arxiv.1809.02922},
  eprint = {1809.02922},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1809.02922}
}

References

38 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Bahdanau, K. Cho, and Y. Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR).
  2. C. Bannard and C. Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Association for Computational Linguistics (ACL), pages 597–604.
  3. S. Bowman, G. Angeli, C. Potts, and C. D. Manning. 2015. A large annotated corpus for learning natural language inference. In Empirical Methods in Natural Language Processing (EMNLP).
  4. R. Chandrasekar, C. Doran, and B. Srinivas. 1996. Motivations and methods for text simplification. In Proceedings of the 16th conference on Computational linguistics-Volume 2, pages 1041–1044.
  5. H. Chen, S. Huang, D. Chiang, and J. Chen. 2017. Improved neural machine translation with a syntax-aware encoder and decoder. arXiv preprint arXiv:1707.05436.
  6. A. Conneau, D. Kiela, H. Schwenk, L. Barrault, and A. Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364.
  7. I. Dagan, O. Glickman, and B. Magnini. 2006. The PASCAL recognising textual entailment challenge. Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment. find
  8. T. Dozat, P. Qi, and C. D. Manning. 2017. Stanford’s graph-based neural dependency parser at the conll 2017 shared task. In Computational Natural Language Learning (CoNLL), pages 20–30.
  9. A. Eriguchi, Y. Tsuruoka, and K. Cho. 2017. Learning to parse and translate improves neural machine translation. arXiv preprint arXiv:1702.03525.
  10. J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin. 2017. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122.
  11. A. Graves. 2012. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211.3711.
  12. J. Gu, Z. Lu, H. Li, and V. O. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In Association for Computational Linguistics (ACL).
  13. S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. R. Bowman, and N. A. Smith. 2018. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324.
  14. L. He, M. Lewis, and L. Zettlemoyer. 2015. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In Empirical Methods in Natural Language Processing (EMNLP).
  15. S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. find
  16. K. S. Jones. 1993. What might be in a summary? Information retrieval, 93:9–26. find
  17. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics (ACL).
  18. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Proceedings of AAAI. find
  19. D. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  20. V. Kolesnyk, T. Rocktäschel, and S. Riedel. 2016. Generating natural language inference chains. arXiv preprint arXiv:1606.01404. — [arXiv:1606.01404](https://arxiv.org/abs/1606.01404) find
  21. A. Lai, Y. Bisk, and J. Hockenmaier. 2017a. Natural language inference from multiple premises. arXiv preprint arXiv:1710.02925.
  22. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. 2017b. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683.
  23. D. Lin and P. Pantel. 2001. Discovery of inference rules for question-answering. Natural Language Engineering, 7:343–360. find
  24. M. Marelli, S. Menini, M. Baroni, L. Bentivogli, R. bernardi, and R. Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Language Resources and Evaluation Conference (LREC).
  25. J. Michael, G. Stanovsky, L. He, I. Dagan, and L. Zettlemoyer. 2018. Crowdsourcing question– answer meaning representations. In North American Association for Computational Linguistics (NAACL).
  26. E. Pavlick and C. Callison-Burch. 2016. Most” babies” are” little” and most” problems” are” huge”: Compositional entailment in adjective-nouns. In Association for Computational Linguistics (ACL), volume 1, pages 2164–2173. find
  27. J. Pennington, R. Socher, and C. D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP).
  28. Adam Poliak, Aparajita Haldar, Rachel Rudinger, J Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. 2018. Towards a unified natural language inference framework to evaluate sentence representations. arXiv preprint arXiv:1804.08207.
  29. P. Rajpurkar, R. Jia, and P. Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Association for Computational Linguistics (ACL).
  30. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP).
  31. M. Richardson, C. J. Burges, and E. Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP), pages 193–203.
  32. V. Rus and C. G. Arthur. 2009. The question generation shared task and evaluation challenge. In The University of Memphis. National Science Foundation. find
  33. M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler. 2016. Movieqa: Understanding stories in movies through question-answering. In Computer Vision and Pattern Recognition (CVPR), pages 4631–4640. find
  34. A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman. 2017. NewsQA: A machine comprehension dataset. In Workshop on Representation Learning for NLP.
  35. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762.
  36. Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. Inference is everything: Recasting semantic resources into a unified evaluation framework. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pa…
  37. A. Williams, N. Nangia, and S. R. Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426.
  38. L. Yu, J. Buys, and P. Blunsom. 2016. Online segment to segment neural transduction. arXiv preprint arXiv:1609.08194. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/demszky-2018-transforming-question-answering-datasets-natural/