Elamur Research⌕
☰
home / papers / 2018 / Efficient and Robust Question Answering from Minimal Context over Documents

Efficient and Robust Question Answering from Minimal Context over Documents

Paper · 2018 · Sewon Min, Victor Zhong, Richard Socher et al.

Summary

Neural reading-comprehension models that jointly encode a question and a whole document, for example through coattention, become slow on long or multi-document inputs. They are also easily misled by distracting adversarial sentences. Min et al. first measure how much text a question really needs. Manual inspection of samples from SQuAD and TriviaQA (Wikipedia) shows that one or two sentences usually suffice. DCN+ trained only on the answer-bearing sentence even slightly beats its full-document version.

Building on this, the authors add a lightweight sentence selector that shares its encoder with the S-Reader QA model. It scores all sentences in parallel and forwards a variable, question-dependent number of them to the reader. Three techniques make the selector effective: weight transfer, data modification and score normalization.

On SQuAD, NewsQA, TriviaQA (Wikipedia) and SQuAD-Open, this MINIMAL setup matches or exceeds the FULL baseline while training and inferring much faster. It sets new state-of-the-art results on three of these datasets. It also resists adversarial distractors far better, showing that pruning context helps both scalability and robustness.

Key points

  • Human study (50 examples per dataset): On SQuAD, 98% of questions are answerable, and 92% of those need just one sentence. TriviaQA documents average 488 sentences versus about 5 for SQuAD, yet 95% of its answerable questions need at most two sentences.
  • Oracle-sentence experiment: DCN+ reaches 83.1 F1 with the full document and 85.1 F1 with only the oracle sentence on the SQuAD dev set. The ORACLE model answers 93% (SQuAD) and 86% (NewsQA) of the questions FULL gets right.
  • Efficiency: MINIMAL gives up to 15× faster training and up to 13× faster inference than FULL. Dynamic sentence selection outperforms TF-IDF retrieval at a similar number of sentences (59.5 vs 51.9 F1).
  • Adversarial robustness (SQuAD-Adversarial dev, top-1 sentence): DCN+ MINIMAL scores 59.7 F1 on AddSent and 67.5 F1 on AddOneSent. This beats the earlier best, Mnemonic Reader (46.6 and 56.0), and improves over FULL by roughly 11 F1 on both sets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Seoul National UniversitySeoul · South Korea20 works · 45,282 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksInformation Systems · 523 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 242 works

Cite

Cite as Sewon Min, Victor Zhong, Richard Socher and Caiming Xiong. 2018. Efficient and Robust Question Answering from Minimal Context over Documents. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1160

@inproceedings{min2018efficient,
  title = {{Efficient and Robust Question Answering from Minimal Context over Documents}},
  author = {Sewon Min, Victor Zhong, Richard Socher and Caiming Xiong},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1160},
  url = {https://doi.org/10.18653/v1/p18-1160}
}

References

40 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In EMNLP.
  2. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In ACL.
  3. Eunsol Choi, Daniel Hewlett, Jakob Uszkoreit, Illia Polosukhin, Alexandre Lacoste, and Jonathan Berant. 2017. Coarse-to-fine question answering for long documents. In ACL. find
  4. Christopher Clark and Matt Gardner. 2017. Simple and effective multi-paragraph reading comprehension. arXiv preprint arXiv:1710.10723 .
  5. Bhuwan Dhingra, Kathryn Mazaitis, and William W Cohen. 2017. Quasar: Datasets for question answering by search and reading. arXiv preprint arXiv:1707.03904 . find
  6. Matthew Dunn, Levent Sagun, Mike Higgins, Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179 .
  7. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple nlp tasks. In EMNLP.
  8. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems.
  9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation . find
  10. Minghao Hu, Yuxing Peng, and Xipeng Qiu. 2017. Mnemonic reader for machine comprehension. arXiv preprint arXiv:1705.02798 . find
  11. Hsin-Yuan Huang, Chenguang Zhu, Yelong Shen, and Weizhu Chen. 2018. Fusionnet: Fusing via fully-aware attention with application to machine comprehension. In ICLR. find
  12. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In EMNLP.
  13. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL.
  14. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In EMNLP.
  15. Kenton Lee, Shimi Salant, Tom Kwiatkowski, Ankur Parikh, Dipanjan Das, and Jonathan Berant. 2016. Learning recurrent span representations for extractive question answering. arXiv preprint arXiv:1611.01436 . — [arXiv:1611.01436](https://arxiv.org/abs/1611.01436) find
  16. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In ACL.
  17. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NIPS.
  18. Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In EMNLP. find
  19. Sewon Min, Minjoon Seo, and Hannaneh Hajishirzi. 2017. Question answering through transfer learning from large fine-grained supervision data. In ACL. find
  20. Boyuan Pan, Hao Li, Zhou Zhao, Bin Cao, Deng Cai, and Xiaofei He. 2017. Memen: Multi-layer embedding with memory networks for machine comprehension. arXiv preprint arXiv:1707.09098 . — [arXiv:1707.09098](https://arxiv.org/abs/1707.09098) find
  21. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
  22. Jonathan Raiman and John Miller. 2017. Globally normalized reader. In EMNLP. find
  23. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP.
  24. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP.
  25. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR.
  26. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1706.02596v2 . find
  27. Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2017. Reasonet: Learning to stop reading in machine comprehension. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. find
  28. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2017. The narrativeqa reading comprehension challenge. arXiv preprint arXiv:1712.07040 .
  29. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research . find
  30. Swabha Swayamdipta, Ankur P Parikh, and Tom Kwiatkowski. 2018. Multi-mention learning for reading comprehension with neural cascades. In ICLR. find
  31. Chuanqi Tan, Furu Wei, Qingyu Zhou, Nan Yang, Bowen Du, Weifeng Lv, and Ming Zhou. 2018. Context-aware answer sentence selection with hierarchical gated recurrent neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing . find
  32. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830 .
  33. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerald Tesauro, Bowen Zhou, and Jing Jiang. 2018. R3: Reinforced reader-ranker for open-domain question answering. In AAAI. find
  34. Dirk Weissenborn. 2017. Reading twice for natural language understanding. CoRR abs/1706.02596. — [arXiv:1706.02596](https://arxiv.org/abs/1706.02596) find
  35. Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Making neural qa as simple as possible but not simpler. In CoNLL. find
  36. Qizhe Xie, Guokun Lai, Zihang Dai, and Eduard Hovy. 2017. Large-scale cloze test dataset designed by teachers. arXiv preprint arXiv:1711.03225 . — [arXiv:1711.03225](https://arxiv.org/abs/1711.03225) find
  37. Caiming Xiong, Victor Zhong, and Richard Socher. 2018. Dcn+: Mixed objective and deep residual coattention for question answering. In ICLR. find
  38. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In EMNLP.
  39. Wenpeng Yin, Hinrich Schtze, Bing Xiang, and Bowen Zhou. 2016. Abcnn: Attention-based convolutional neural network for modeling sentence pairs. TACL .
  40. Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. Fast and accurate reading comprehension by combining self-attention and convolution. In ICLR. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/min-2018-efficient-robust-question-answering-minimal/