Elamur Research⌕
☰
home / papers / 2021 / Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models

Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models

Paper · 2021 · Tushar Khot, Daniel Khashabi, Kyle Richardson et al.

Summary

Text Modular Networks (TMNs) tackle complex question answering by breaking a hard question into a chain of simpler questions that existing models can already answer. Earlier decomposition methods were built for one dataset and produced sub-questions without regard to which solver would receive them. TMNs instead learn how each existing sub-model "speaks." A BART model is trained on a sub-model's own dataset, such as SQuAD, to generate questions from a context, an answer and vocabulary hints. Using distant supervision and no annotated decompositions, the authors then build step-by-step decomposition examples and train a next-question generator. This generator picks a sub-model and writes the next sub-question until it emits an end symbol. The resulting system, MODULARQA, combines a neural single-span QA model with a symbolic calculator. It covers five reasoning types: composition, conjunction, comparison, difference and complementation. It is the first decomposition-based interpretable system that works on both DROP and HotpotQA. It comes close to black-box models in accuracy while being more robust and sample-efficient, and its explanations are faithful and written in natural language.

Key points

  • On DROP, MODULARQA scores only 3.7% F1 below the black-box NumNet+V2. With 20% of the training data it outperforms NumNet+V2 (87.0 vs 85.4 F1), although NumNet+V2 leads at 100% (91.6 vs 87.8).
  • Evaluated on 239 in-scope questions from the contrast set for DROP introduced by Gardner et al. (2020), MODULARQA reaches 55.7 EM and 63.3 F1, compared with 45.2 EM and 56.2 F1 for NumNet+V2.
  • It clearly beats the DROP-specific NMN-D and roughly matches S-NMN on HotpotQA. It trails DecompRC and Quark on HotpotQA, and the authors note that both rely on dataset-specific design or supporting-fact annotations.
  • Human judges rated its explanations as more trustworthy, understandable and preferable than those of the split-based DecompRC in 67%–78% of cases.
  • Error analysis:
    • On HotpotQA, 65% of errors come from intermediate questions that have multiple or yes/no answers.
    • On DROP, errors stem mainly from decomposition (53.3%) and the QA sub-models (33.33%).
    • Code is available at github.com/allenai/modularqa.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States219 works · 89,712 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Vision and Pattern Recognition · 1,282 works

Cite

Cite as Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark and Ashish Sabharwal. 2021. Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.naacl-main.99

@inproceedings{khot2021text,
  title = {{Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models}},
  author = {Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark and Ashish Sabharwal},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.99},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.99}
}

References

43 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural networks for question answering. In NAACL-HLT.
  2. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic Parsing on Freebase from Question-Answer Pairs. In EMNLP.
  3. Gino Brunner, Y. Liu, Damián Pascual, Oliver Richter, and Roger Wattenhofer. 2020. On identifiability in transformers. In ICLR. find
  4. Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke Zettlemoyer, and Eduard Hovy. 2019. Iterative search for weakly supervised semantic parsing. In NAACL. find
  5. Edsger W Dijkstra et al. 1959. A note on two problems in connexion with graphs. Numerische mathematik, 1(1):269–271. find
  6. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL.
  7. Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. 2020. Hierarchical graph network for multi-hop question answering. In EMNLP.
  8. J Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. 2019a. Improved Lexically Constrained Decoding for Translation and Monolingual Rewriting. In NAACL.
  9. Minghao Hu, Yuxing Peng, Zhen Huang, and Dong sheng Li. 2019b. A multi-type multi-span network for reading comprehension that requires discrete reasoning. In EMNLP. find
  10. Alon Jacovi and Y. Goldberg. 2020. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In ACL.
  11. Yichen Jiang and Mohit Bansal. 2019. Self-assembling modular networks for interpretable multi-hop reasoning. In EMNLP-IJCNLP. find
  12. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In NAACL.
  13. Tushar Khot, Peter Clark, Michal Guerquin, Paul Edward Jansen, and Ashish Sabharwal. 2020. QASC: A dataset for question answering via sentence composition. In AAAI. find
  14. Louis Kirsch, Julius Kunze, and David Barber. 2018. Modular networks: Learning to decompose neural computation. In NeurIPS. find
  15. Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noa… find
  16. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL.
  17. Dirk Groeneveld, Tushar Khot, Ashish Sabharwal, et al. 2020. A simple yet strong pipeline for hotpotqa. In EMNLP. find
  18. John McCarthy. 1988. Epistemological challenges for connectionism. Behavioral and Brain Sciences, 11(1):44–44. find
  19. Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. 2020. Neural module networks for reasoning over text. In ICLR. find
  20. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019a. Compositional questions do not necessitate multi-hop reasoning. In ACL.
  21. Sanda Harabagiu and Andrew Hickl. 2006. Methods for using textual entailment in open-domain question answering. In ACL.
  22. Chris Hokamp and Qun Liu. 2017. Lexically Constrained Decoding for Sequence Generation using Grid Beam Search. In ACL.
  23. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In ICLR. find
  24. Percy Liang, Michael I Jordan, and Dan Klein. 2013. Learning dependency-based compositional semantics. Computational Linguistics, 39(2):389–446. find
  25. Sewon Min, Victor Zhong, Luke S. Zettlemoyer, and Hannaneh Hajishirzi. 2019b. Multi-hop reading comprehension through question decomposition and rescoring. In ACL.
  26. Dan I. Moldovan, Sanda M. Harabagiu, Marius Pasca, Rada Mihalcea, Roxana Girju, Richard Goodrum, and Vasile Rus. 2000. The structure and performance of an open-domain question answering system. In ACL. find
  27. Arvind Neelakantan, Quoc V Le, Martin Abadi, Andrew McCallum, and Dario Amodei. 2017. Learning a natural language interface with neural programmer. In ICLR. find
  28. Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understanding benchmark. TACL, 8:183–198.
  29. Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. Unsupervised question decomposition for question answering. In EMNLP.
  30. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP.
  31. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP.
  32. Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. 2019. Numnet: Machine reading comprehension with numerical reasoning. In EMNLP-IJCNLP. find
  33. Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. 2018. Routing networks: Adaptive selection of non-linear functions for multi-task learning. In ICLR. find
  34. Subhro Roy and Dan Roth. 2018. Mapping to declarative knowledge for word problem solving. TACL, 6:159–172.
  35. Elad Segal, Avia Efrat, Mor Shoham, Amir Globerson, and Jonathan Berant. 2020. A simple and effective model for answering multi-span questions. In EMNLP. find
  36. Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. 2015. Solving geometry problems: Combining text and diagram interpretation. In EMNLP.
  37. Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In ACL.
  38. Paul Smolensky. 1988. On the proper treatment of connectionism. Behavioral and brain sciences, 11(1):1– 23. find
  39. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In NAACL.
  40. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2020. Is multihop qa in dire condition? measuring and reducing disconnected reasoning. In EMNLP. find
  41. Ming Tu, Kevin Huang, Guangtao Wang, Jui-Ting Huang, Xiaodong He, and Bufang Zhou. 2020. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. In AAAI. find
  42. Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In EMNLP/IJCNLP.
  43. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229–256. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/khot-2021-text-modular-networks-learning-decompose/