Elamur Research⌕
☰
home / papers / 2019 / Finding Generalizable Evidence by Learning to Convince Q&A Models

Finding Generalizable Evidence by Learning to Convince Q&A Models

Paper · 2019 · Ethan Perez, Siddharth Karamcheti, Rob Fergus et al.

Summary

The paper asks whether the passage sentences that best support an answer can be found automatically, and whether that evidence is useful beyond the model that produced it. The authors use multiple-choice reading comprehension (RACE and DREAM) as a testbed. Each answer option gets its own "evidence agent," which picks passage sentences that raise a pretrained QA "judge" model's probability for that option. The judge sees only the selected sentences, not the whole passage. Judges include TFIDF and fastText similarity baselines plus fine-tuned BERT BASE and BERT LARGE. Agents either search exhaustively or are learned BERT BASE predictors of the judge's response, and they can act alone or compete (free-for-all or round robin). Evidence chosen to convince one judge also shifts other models and human readers toward the supported answer. Humans keep most of their accuracy while reading about a fifth of the passage, and QA models generalize better to longer passages and harder questions. This points to model-guided evidence selection as a practical aid for reading and robust QA.

Key points

  • Datasets: RACE has about 100K written passages with 4 answer options per question and is split into middle- and high-school exams. DREAM has about 10K dialogue passages with 3 options per question.
  • Judge accuracy on full passages: BERT LARGE reaches 69.4% on RACE and 64.9% on DREAM. The TFIDF and fastText judges fall far lower, roughly 30–45%.
  • Human study: People answered from agent-selected sentences only, about 3.3 vs 18.2 sentences on RACE and 2.4 vs 12.2 on DREAM. Their accuracy was 73.2% vs 82.3% (RACE) and 83.8% vs 93.0% (DREAM), keeping roughly 90% of full-passage accuracy.
  • Generalization: Answering from the most convincing evidence helps judges transfer from short to long passages, and from middle-school to high-school RACE questions.
  • Learned agents: Training an agent only requires querying the judge, not gold answer labels. Agents can therefore be trained on unlabeled or out-of-distribution data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States183 works · 228,558 citations🎓New York UniversityNew York · United States201 works · 100,180 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela and Kyunghyun Cho. 2019. Finding Generalizable Evidence by Learning to Convince Q&A Models. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1244 · arXiv 1909.05863

@inproceedings{perez2019finding,
  title = {{Finding Generalizable Evidence by Learning to Convince Q\&A Models}},
  author = {Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela and Kyunghyun Cho},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1244},
  eprint = {1909.05863},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1244}
}

References

31 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. 2019. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR. find
  2. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In EMNLP, pages 2890–2896.
  3. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In NAACL, pages 107–112.
  4. Kristjan Arumae and Fei Liu. 2018. Reinforced extractive summarization with question-focused rewards. find
  5. Geoffrey Irving and Amanda Askell. 2019. AI safety needs social scientists. Distill. find
  6. Geoffrey Irving, Paul Christiano, and Dario Amodei. 2018. AI safety via debate. CoRR, abs/1805.00899. find
  7. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In EMNLP, pages 2021–2031.
  8. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In EACL, pages 427–431. find
  9. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
  10. Bernhard Kratzwald and Stefan Feuerriegel. 2019. Learning from on-line user feedback in neural question answering on the web. In WWW, pages 906– 916. find
  11. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding Comprehension dataset from Examinations. In EMNLP, pages 785–794.
  12. Barbara Landau, Linda B. Smith, and Susan S. Jones. 1988. The importance of shape in early lexical learning. Cognitive Development, 3(3):299–321. find
  13. Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In EMNLP, pages 107–117.
  14. Jiwei Li, Will Monroe, and Daniel Jurafsky. 2016. Understanding neural networks through representation erasure. CoRR, abs/1612.08220. find
  15. Alexander H. Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. 2017. Parlai: A dialog research software platform. CoRR, abs/1705.06476.
  16. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. In NIPS-W. find
  17. Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. 2018. Film: Visual reasoning with a general conditioning layer. In AAAI, pages 3942–3951.
  18. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In ACL, pages 856–865. find
  19. M. Schuster and K. Nakajima. 2012. Japanese and korean voice search. In ICASSP, pages 5149–5152. find
  20. Chenglei Si. 2019. Bert for multiple choice machine comprehension. find
  21. Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What makes reading comprehension questions easier? In EMNLP, pages 4208– 4219.
  22. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. Dream: A challenge dataset and models for dialogue-based reading comprehension. TACL, 7:217–231. find
  23. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. CoRR, abs/1312.6199.
  24. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In NAACL, pages 809–819.
  25. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, NeurIPS, pages 5998–6008. Curran Asso…
  26. Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, Dong Yu, Dan Roth, and David McAllester. 2018a. Evidence extraction for machine reading comprehension with deep probabilistic logic. CoRR, abs/1902.08852. — [arXiv:1902.08852](https://arxiv.org/abs/1902.08852) find
  27. Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2018b. Evidence aggregation for answer re-ranking in open-domain question answering. In ICLR. find
  28. Mo Yu, Shiyu Chang, and Tommi S Jaakkola. 2019. Learning corresponded rationales for text matching. find
  29. Christian S. Perone, Roberto Silveira, and Thomas S. Paula. 2018. Evaluation of sentence embeddings in downstream and linguistic probing tasks. CoRR, abs/1806.06259. — [arXiv:1806.06259](https://arxiv.org/abs/1806.06259) find
  30. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  31. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, pages 2383–2392.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/perez-2019-finding-generalizable-evidence-learning-convince/