Elamur Research⌕
☰
home / papers / 2020 / Unsupervised Question Decomposition for Question Answering

Unsupervised Question Decomposition for Question Answering

Summary

Multi-hop questions are hard for QA systems because the evidence they need is spread across several passages, while simpler single-hop questions are handled well. Earlier decomposition methods, such as DecompRC and IBM Watson, depend on human annotations or hand-written rules. This paper presents ONUS (One-to-N Unsupervised Sequence transduction), which learns to split a complex question into simpler sub-questions with no labeled decompositions. The authors first build noisy pseudo-decompositions by retrieving similar but mutually diverse simple questions using FastText-based embeddings. They then train a transformer on this data with unsupervised sequence-to-sequence learning. An off-the-shelf single-hop QA model answers each sub-question. A recomposition model then combines the sub-questions, their answers and the original context into the final prediction. On HotpotQA, the approach clearly beats a strong RoBERTa baseline, especially on the out-of-domain and multi-hop challenge sets. It is competitive with SAE and HGN, which need supporting-fact supervision. The generated sub-questions are also more fluent than heuristic ones and make model predictions easier to interpret.

Key points

  • The training corpora were expanded by mining Common Crawl with a FastText classifier. Single-hop questions grew from about 130K (SQuAD 2) to 10.1M, and multi-hop questions grew from about 90K (HotpotQA) to 2.4M. The pseudo-decompositions use a fixed N = 2 sub-questions.
  • Using decompositions improves F1 over the RoBERTa baseline by 3.1 points on the original HotpotQA dev set. The improvement rises to 10 points on Min et al.'s (2019b) out-of-domain dev set and to 11 points on Jiang and Bansal's (2019a) multi-hop dev set.
  • The system is robust to the choice of single-hop answerer. A BERT-base ensemble scores far below the RoBERTa-large ensemble on HotpotQA directly (56.3 vs. 66.7 F1), yet the full pipeline performs almost the same with either (79.9 vs. 80.1 F1).
  • The benefit of decompositions grows with the strength of the recomposition model: +1.2 F1 with BERT-base, +2.6 with BERT-large and +3.1 with RoBERTa-large.
  • ONUS covers all four HotpotQA question types with a single model. Without fine-tuning, it can also decompose some visual QA and knowledge-base QA questions, as well as FEVER claims. Code, data and models are released on GitHub (facebookresearch/UnsupervisedDecomposition).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

98 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 88 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.713

@inproceedings{perez2020unsupervised,
  title = {{Unsupervised Question Decomposition for Question Answering}},
  author = {Perez, Ethan and Lewis, Patrick and Yih, Wen-tau and Cho, Kyunghyun and Kiela, Douwe},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.713},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.713}
}

References

52 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2015. Neural module networks. CVPR, pages 39–48. find our card
  2. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural networks for question answering. In NAACL, pages 1545–1554, San Diego, California. Association for Computational Linguistics. find our card
  3. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In ICLR. find our card
  4. Mikel Artetxe and Holger Schwenk. 2019. Margin-based parallel corpus mining with multilingual sentence embeddings. In ACL, pages 3197–3203, Florence, Italy. Association for Computational Linguistics. find our card
  5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL, 5:135–146. find
  6. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In ACL, pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics. find our card
  7. Paul Francis Christiano, Buck Shlegeris, and Dario Amodei. 2018. Supervising strong learners by amplifying weak experts. CoRR, abs/1810.08575. find
  8. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In ACL, pages 845–855, Melbourne, Australia. Association for Computational Linguistics. find our card
  9. Abhishek Das, Satwik Kottur, José M.F. Moura, Stefan Lee, and Dhruv Batra. 2017. Learning cooperative visual dialog agents with deep reinforcement learning. In ICCV. find
  10. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017c. Inferring and executing programs for visual reasoning. In ICCV. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186. find
  12. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In EACL, pages 427–431, Valencia, Spain. Association for Computational Linguistics. find
  13. Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In ACL, pages 33– 43, Berlin, Germany. Association for Computational Linguistics. find
  14. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. In NeurIPS. find
  15. Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. 2019. Hierarchical graph network for multi-hop question answering. find
  16. Manaal Faruqui and Dipanjan Das. 2018. Identifying well-formed natural language questions. In EMNLP, pages 798–803, Brussels, Belgium. Association for Computational Linguistics. find
  17. David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A. Kalyanpur, Adam Lally, J. William Murdock, Eric Nyberg, John Prager, Nico Schlaefer, and Chris Welty. 2010. Building watson: An overview of the deepqa project. AI Magazine, 31(3):59–79. find
  18. Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. TACL, 6:437–450. find our card
  19. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. find
  20. Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. 2017. Reinforcement learning with unsupervised auxiliary tasks. In ICLR. find
  21. Yichen Jiang and Mohit Bansal. 2019a. Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop QA. In ACL, pages 2726–2736, Florence, Italy. Association for Computational Linguistics. find
  22. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Unsupervised machine translation using monolingual corpora only. In ICLR. find our card
  23. Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel. 2019. Unsupervised question answering by cloze translation. In ACL, pages 4896–4910, Florence, Italy. Association for Computational Linguistics. find
  24. Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le, and Ni Lao. 2018. Memory augmented policy optimization for program synthesis and semantic parsing. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, NeurIPS, pages 9994–10006. Curran Associates, In… find
  25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. CoRR, abs/1907.11692. find
  26. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed precision training. In ICLR. find
  27. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019a. Compositional questions do not necessitate multi-hop reasoning. In ACL, pages 4249–4257, Florence, Italy. Association for Computational Linguistics. find our card
  28. Yichen Jiang and Mohit Bansal. 2019b. Self-assembling modular networks for interpretable multi-hop reasoning. In EMNLP, Hong Kong, China. Association for Computational Linguistics. find
  29. Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019b. Multi-hop reading comprehension through question decomposition and rescoring. In ACL, pages 6097–6109, Florence, Italy. Association for Computational Linguistics. find our card
  30. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017a. Billion-scale similarity search with gpus. CoRR, abs/1702.08734. find our card
  31. Yixin Nie, Songhe Wang, and Mohit Bansal. 2019. Revealing the importance of semantic retrieval for machine reading at scale. In EMNLP. find
  32. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. 2017b. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR. find our card
  33. Rodrigo Nogueira and Kyunghyun Cho. 2017. Task-oriented query reformulation with reinforcement learning. In EMNLP, pages 574–583, Copenhagen, Denmark. Association for Computational Linguistics. find
  34. Michael Petrochuk and Luke Zettlemoyer. 2018. SimpleQuestions nearly solved: A new upperbound and baseline approach. In EMNLP, pages 554–558, Brussels, Belgium. Association for Computational Linguistics. find
  35. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  36. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In ACL, pages 86–96, Berlin, Germany. Association for Computational Linguistics. find
  37. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771. find our card
  38. Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understanding benchmark. TACL. find our card
  39. W. Woods, R. Kaplan, and B. Nash-Webber. 1974. The lunar sciences natural language information system. Final Report 2378, Bolt, Beranek and Newman, Inc., Cambridge, MA. find
  40. Florian Strub, Harm de Vries, Jérémie Mary, Bilal Piot, Aaron Courville, and Olivier Pietquin. 2017. End-to-end optimization of goal-driven and visually grounded dialogue systems. In IJCAI, pages 2765– 2771. find
  41. Hainan Xu and Philipp Koehn. 2017. Zipporah: a fast and scalable data cleaning system for noisy web-crawled parallel corpora. In EMNLP, pages 2945– 2950, Copenhagen, Denmark. Association for Computational Linguistics. find
  42. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In NAACL, pages 641–651, New Orleans, Louisiana. Association for Computational Linguistics. find
  43. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics. find our card
  44. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. CoRR, abs/1803.05355. find our card
  45. James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2019. The FEVER2.0 shared task. In Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER), pages 1–6, Hong Kong, China. Association for Computational Linguistics. find
  46. Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xiaodong He, and Bowen Zhou. 2020. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. In AAAI. find
  47. Guangyou Zhou, Tingting He, Jun Zhao, and Po Hu. 2015. Learning continuous word embedding with metadata for question retrieval in community question answering. In ACL, pages 250–259, Beijing, China. Association for Computational Linguistics. find
  48. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, page 19–27, USA. IEEE Computer Society. find our card
  49. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, NeurIPS, pages 5998–6008. Curran Asso… find
  50. Ziyun Wang and Brenden M. Lake. 2019. Modeling question asking using neural program generation. CoRR, abs/1907.09899. — [arXiv:1907.09899](https://arxiv.org/abs/1907.09899) find
  51. Terry Winograd. 1972. Understanding Natural Language. Academic Press, Inc., USA. find
  52. Terry Winograd. 1991. Thinking Machines: Can There Be? Are We? University of California Press, Berkeley. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/perez-2020-unsupervised-question-decomposition-question-answering/